SQS Is Delivering the Same Message Twice. Is the Queue Broken?
Scenario
A job-processing system built on a standard SQS queue is occasionally running the same job twice. Customers get two copies of the same email; a few get charged twice. The workers pull messages, spend roughly two minutes doing the work, and delete the message when they finish. The queue's visibility timeout is the default-ish 30 seconds. There are no idempotency keys anywhere in the pipeline, and no dead-letter queue. The duplicates are intermittent, which makes them feel random — and random feels like a platform bug.
The Quick Fix on the Table
A teammate has concluded that standard SQS is simply unreliable and wants to migrate everything to FIFO queues, where "exactly-once processing" is on the label. It sounds like a clean fix: swap the queue type, duplicates disappear, no application code changes.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- The queue isn't broken — the timeout is. A 30-second visibility timeout on a 2-minute job means every message becomes visible again while it's still being worked on, and a second consumer legitimately picks it up. This is documented at-least-once behavior, not a defect.
- FIFO doesn't fix this failure mode. FIFO deduplication only suppresses duplicate sends within a 5-minute window. A message redelivered because its visibility timeout expired mid-processing will be redelivered on FIFO too. The migration would ship the same bug with a new queue type.
- "Exactly-once" ends at the queue boundary. Even a perfect queue can't make a charge-the-customer side effect exactly-once: the worker can crash after charging but before deleting the message. Duplicates can always reach your code; only your code can make them harmless.
- FIFO adds real constraints for nothing. Lower throughput ceilings, message-group ordering semantics, and a disruptive migration — all paid in exchange for not fixing the problem.
- The blame frame kills learning. Declaring the platform broken ends the investigation exactly where it should begin, and the team walks away not understanding delivery semantics — until the next distributed-systems surprise.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Confirm the mechanism. Check the
ApproximateReceiveCountattribute on processed messages and correlate duplicate incidents with job duration. Jobs longer than 30 seconds with receive counts > 1 confirm visibility-timeout expiry, not queue malfunction. - Size the visibility timeout to the work. Set it comfortably above worst-case processing time — for a ~2-minute job, something like 6 minutes. For jobs with high variance, have workers call
ChangeMessageVisibilityas a heartbeat to extend the lease while still processing. - Make consumers idempotent — this is the core fix. Derive an idempotency key (order ID, message ID, or a producer-set business key) and record it in a store with conditional writes — e.g. DynamoDB
PutItemwithattribute_not_exists, or a Postgres unique constraint. If the key already exists, acknowledge and skip. - Make the side effects idempotent too. Pass the idempotency key downstream: payment providers accept idempotency keys on charge requests, and email sends can be deduplicated on the same key. A duplicate delivery then costs a no-op, not a double charge.
- Add a dead-letter queue with an alarm. Configure a redrive policy (e.g.
maxReceiveCountof 5) so poison messages stop cycling forever, and alert on DLQ depth so failures are seen instead of silently retried. - Delete promptly and monitor. Delete the message immediately after the transaction commits, and track duplicate-skip counts as a metric — a healthy system shows a small nonzero rate, proving the guard is doing its job.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Standard SQS promises at-least-once delivery and it's keeping that promise. Our 30-second lease on a 2-minute job is the bug — the queue is doing exactly what the docs say."
- "FIFO dedup only covers duplicate publishes within five minutes. Our duplicates come from redelivery during processing, which FIFO will happily do too — we'd migrate everything and keep the double charges."
- "No queue technology can make charging a customer exactly-once, because the worker can die between the charge and the delete. Idempotency in the consumer is the only fix that holds."
- "The fix is two config changes and one conditional write — days, not the weeks a queue migration costs, and it protects us against every future source of duplicates, not just this one."
- "Idempotent consumers also unlock safe operational retries: we can redrive the DLQ or replay messages without fear, which we can't do today regardless of queue type."