queues · Sep 10, 2026
Moving a Queue Without Stopping It
Four phases, one rollback point each, and a consumer that doesn't mind seeing the same message twice. How we moved a production job queue to a new broker with no maintenance window.

We had a job queue that handled everything from receipts to nightly exports, and we needed to move it to a different broker. The old one was out of support and increasingly cranky. The constraint, handed down by the people who answer the support phone, was no maintenance window. Receipts don't stop for infrastructure.
Kasimir drew the plan on a whiteboard in four boxes. We followed it almost exactly, which has never happened to me before and probably won't again.
The four phases
Old only. Everything as it is today. The new broker exists and is empty.
Dual write. Producers send every message to both brokers. Consumers still read only from the old one. Nothing changes for users, and we can compare the two sides for as long as we like.
Dual write, read from the new one. We flip the consumers. Producers keep writing to both, so flipping back is a config change, not an emergency.
New only. Producers stop writing to the old broker. It stays running, idle, for a week.
Each step has exactly one thing that can go wrong, and a rollback that takes one deploy. That's the property I'd defend in any migration plan: if a step needs two simultaneous changes, split it.
The consumer is the hard part
Phase three has a quiet trap. Messages written during phase two are sitting in the new broker, and the consumers have already handled them via the old one. When the consumers flip, they'll see those messages a second time.
You can fight that with careful draining and timing, or you can make the consumer indifferent to duplicates. We chose indifference. Every message carries an id, and the consumer keeps a short-lived record of the ids it has finished. A repeat is skipped.
This small simulation walks through the four phases with three messages each.
const queues = { old: [], next: [] };
const names = ["old only", "dual write", "dual write, read next", "next only"];
const publish = (id, phase) => {
if (phase < 3) queues.old.push(id);
if (phase > 0) queues.next.push(id);
};
const consume = (phase) => queues[phase < 2 ? "old" : "next"].splice(0);
const seen = new Set();
let duplicates = 0;
names.forEach((name, phase) => {
for (let i = 0; i < 3; i++) publish(`m${phase}${i}`, phase);
for (const id of consume(phase)) seen.has(id) ? duplicates++ : seen.add(id);
console.log(`${name.padEnd(22)} unique ${seen.size} duplicates ${duplicates}`);
});The last line it prints reads twelve unique messages and three duplicates. Nothing lost, and the three repeats, all from phase two, were absorbed by the id check. That is the whole argument for the design in fourteen lines, though the real consumer's record lives in a database and survives restarts.
What the real run looked like
Phase two ran for six days. We diffed message counts every hour and found one real discrepancy: a batch producer that built its own client and had never been told about the second broker. We'd have lost its messages at cutover. Finding it cost one afternoon, and not finding it would have cost a customer's month-end export.
The flip in phase three took four minutes and nobody outside the team noticed. Phase four was a config change and a quiet Friday.
What I'd do again, and what I wouldn't
I'd do the dual-write phase again, and I'd make it longer. Six days caught the batch producer. A month would have caught the monthly report job, which fell outside our six days and which we only remembered when the old broker was finally decommissioned and the job failed with a clear, polite error.
I wouldn't trust the whiteboard for the inventory of producers. Grep the codebase, then grep the cron files, then ask around.