The Queue Backlog Doubles Every Day. 10x the Consumers Tonight?
Scenario
The order-processing queue has gone from 20k to 40k to 80k to 160k messages over three days, and customers are asking where their confirmation emails went. Twenty consumers work the queue, handling payments, inventory reservation, and emails. A deploy touched the inventory-reservation code right around when the growth started. The consumers share the main application's Postgres database, and the payment provider enforces per-account rate limits. Nobody has looked at per-message processing time, redelivery counts, or downstream latency — and the dead-letter queue exists but has no alarm on it. Incoming order volume, notably, is normal.
The Quick Fix on the Table
"The consumers are stateless — set the autoscaling group to 200 tonight and the backlog is gone by morning." Ten times the workers, ten times the throughput. It's one number in one config, and it feels like decisive action.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- The math already says consumers aren't the bottleneck. Order volume is flat but the backlog doubles daily — so throughput fell, right when the inventory-reservation deploy landed. Twenty consumers handled this load fine last week; the interesting question is what changed, not how many workers there are.
- Consumers are stateless; their dependencies are not. Every consumer holds Postgres connections and calls a rate-limited payment API. If workers are slow because they're waiting on the shared database or throttled by the payment provider, 200 of them means 10x the connection pressure and 10x the throttled calls — the per-message time gets worse.
- It can take production down with it. The consumers share Postgres with the main application. Exhausting connections or saturating locks at 3 a.m. converts "late confirmation emails" into "the whole product is down," with the on-call asleep.
- Redelivery may be silently multiplying the work. If the new code makes messages fail or time out, each order is processed multiple times. More consumers churn through more failing attempts faster — and with no DLQ alarm, nobody would even see it happening.
- It's a blind bet made in the dark. Nobody has looked at a single per-consumer metric. Scaling 10x without knowing where time is spent isn't a fix; it's a coin flip with the database as the stake.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Get three numbers first (this takes an hour, not a week). Per-message processing time now vs. before the deploy; redelivery/receive counts on messages; and latency of each downstream call (Postgres queries, payment API, email provider) from logs or a quick timing wrapper. These three numbers locate the bottleneck.
- Interrogate the suspect deploy. The inventory-reservation change lines up with the backlog's start. Diff it for new queries, missing indexes, lock-taking transactions, added synchronous calls, or retry loops. Be ready to roll it back — if throughput recovers, you've found root cause and the backlog drains with the consumers you already have.
- Check the shared database's view.
pg_stat_activityfor waiting/idle-in-transaction sessions, lock contention, and connection counts againstmax_connections. If consumers are queuing on Postgres, the ceiling is the database — and the consumer count is irrelevant until that's fixed. - Alarm the DLQ and inspect it now. Put an alert on dead-letter depth and sample its contents. Poison messages and systematic failures explain both slow throughput and missing emails, and they're invisible today.
- Scale deliberately once the constraint is known. If measurement shows consumers genuinely CPU/IO-bound and downstreams healthy: scale 20 → 40, watch queue depth, downstream latency, and DB connections for 30 minutes, then step again. Stop at the first sign of downstream stress. Stepwise scaling finds the safe ceiling; 10x overnight discovers it by breaking things.
- Protect the customer promise separately. If confirmations matter most, consider prioritizing email sends or a separate fast lane for them while the backlog drains — the business pain and the queue depth are related but not identical problems.
- Leave the lights on afterwards. Dashboards for queue depth, oldest-message age, per-message duration, redelivery rate, and DLQ depth, with alerts on trends. A backlog that doubles daily should page someone on day one, not surprise the team on day three.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Order volume is flat and the backlog is doubling — throughput dropped, and it dropped when the inventory deploy shipped. The fix for a regression is finding it, not multiplying workers around it."
- "The consumers are stateless, but Postgres and the payment provider aren't. If workers are waiting on those, 200 consumers just means 200 things waiting — and possibly a database outage for the main app."
- "I'm asking for one hour of measurement before we act: per-message time, redelivery counts, downstream latency. That hour tells us whether scaling helps, does nothing, or takes production down."
- "If redeliveries are up, part of this backlog is the same orders failing over and over. Scaling that up burns money to fail faster — and risks more double-processing, which for payments is worse than lateness."
- "I'll happily scale — stepwise, 20 to 40 to 80, watching the database and the rate limits at each step. If the metrics stay healthy we keep going; blind 10x at night with a shared DB is how a backlog becomes an outage."