DynamoDB Is Throttling but Capacity Looks Idle. Crank It Up?
Scenario
Your application is getting ProvisionedThroughputExceededException errors from a DynamoDB table, yet the CloudWatch capacity graphs look almost embarrassing: consumed read and write units sit well below what the table has provisioned. The table was sized generously months ago, and on paper there is plenty of headroom. Digging into the access pattern reveals the real story — the partition key has only a handful of distinct values, and one of them (a big tenant, a default category, a "hot" device type) receives the overwhelming majority of traffic.
In the incident channel, a teammate proposes the obvious lever: "Double the provisioned capacity — or just flip the table to on-demand so it scales itself." Throttling plus low utilization looks like an under-provisioning problem, so more capacity should fix it. Right?
The Quick Fix on the Table
Raise provisioned capacity, or switch the table to on-demand mode. Either is a console click or a one-line Terraform change, requires no application changes, and comes with the comforting feeling of having "done something" during an incident.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Throttling happens per partition, not per table. DynamoDB spreads your provisioned throughput across physical partitions, and each partition has a hard ceiling (roughly 3,000 RCU and 1,000 WCU). A single hot key can saturate its partition while the table-level metric — an average across all partitions — looks nearly idle.
- More capacity mostly feeds the cold partitions. When you double table capacity, the extra throughput is distributed across partitions your traffic barely touches. The hot partition's per-partition ceiling does not move, so the throttles continue while the bill grows.
- On-demand hits the same wall. On-demand mode removes table-level capacity planning, but the per-partition throughput limits still apply. A skewed key distribution throttles in on-demand mode just as it does in provisioned mode.
- Adaptive capacity is a cushion, not a cure. DynamoDB's adaptive capacity can shift unused throughput toward hot partitions and even isolate frequently accessed items, but it cannot push a single key past the physical partition maximum. Sustained skew on one key value will always win.
- It teaches the wrong lesson. If the team concludes "throttling means add capacity," the same incident returns at the next traffic step-up — with a bigger invoice attached each time.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Prove the skew before arguing about it. Enable CloudWatch Contributor Insights on the table (
aws dynamodb update-contributor-insights) to see the most-accessed partition keys, and check theThrottledRequestsmetric split by operation. If one key dominates the top-contributors list while table utilization is low, you have your diagnosis in writing. - Classify the hot key. Is it a design flaw (partition key with few distinct values, like
statusorcountry), a legitimate whale (one tenant that is 80% of traffic), or a time-series pattern (everyone writing to today's date)? The remedy differs for each. - Redesign the partition key for spread. The partition key should be the highest-cardinality attribute that your access pattern can query by —
tenant_id,user_id,device_id— not a low-cardinality classifier. Move classifiers into the sort key or a GSI. - Shard the writes when one key is legitimately hot. Suffix the key with a bounded random or calculated shard (
ORDER#2026-07-08#3with shards 0–9) so writes fan out across N partitions. Reads either scatter-gather across the N suffixes or aggregate through a DynamoDB Stream into a summary item. - Absorb read heat with caching. For read-hot keys, put DAX or an application-level cache in front of the table; a hot item that is 99% cache-hits stops being a partition problem at all.
- Smooth bursts at the edge. If the skew is spiky rather than sustained, an SQS queue in front of the writes plus exponential backoff with jitter in the SDK config turns a throttling cliff into a short delay.
- Then revisit capacity mode with real numbers. Once traffic is spread, pick provisioned-with-autoscaling or on-demand based on the measured pattern — as a cost decision, not a firefighting one.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The table isn't out of capacity — one partition is. Doubling capacity spreads the new throughput across partitions we aren't using, so we'd pay roughly twice as much to keep the same throttles."
- "On-demand doesn't repeal the per-partition limits. If 90% of traffic lands on one key, it will throttle in on-demand mode too — we'd just be surprised by it later, with a variable bill."
- "Contributor Insights will show us the hot keys in an hour. Let's spend that hour confirming the diagnosis instead of a sprint absorbing a capacity increase that can't work."
- "Key design is the one fix that scales with us. Write sharding or a higher-cardinality partition key removes the ceiling permanently; capacity increases just rent us distance from it."
- "I'd still add backoff and jitter today as a safety net — that's cheap and correct regardless. But the durable fix is making the traffic spread match how DynamoDB actually partitions data."