Scaling from 1K to 10K requests per second is not a bigger version of the same problem — it is a different problem. The architecture that had run quietly for a year did not degrade gracefully at 10x; it failed in modes our dashboards had never shown before.

What worked at 1K RPS:

  • Synchronous service-to-service calls with default timeouts
  • Database joins for complex queries
  • Full debug logs to CloudWatch, no sampling
  • Simple round-robin load balancing

The first break: connections

At 1K RPS we held roughly 120 database connections against a PostgreSQL max_connections of 500 — comfortable. Past 4K RPS the autoscaler tripled pod count, every new pod opened its own 20-connection pool, and we hit the ceiling within minutes. Queries queued, latency climbed, and everything upstream started timing out. That failure mode is common enough that it gets its own write-up in the autoscaler that killed our database.

The second break: cascading timeouts

Our call chain was API → orders → pricing → inventory, each hop with a 30-second default timeout and three retries. When inventory slowed down, every hop above it retried, roughly tripling traffic per level — a retry storm that pushed p99 latency from 80 ms to over 10 seconds while worker threads sat blocked, waiting on a dependency that was already drowning.

The third break: the observability bill

Log volume scales linearly with traffic: 10x requests meant 10x CloudWatch ingestion, and logging quietly became our second-largest infrastructure line item. We moved to sampling debug-level logs at 1% — right for us because at 10K RPS that still yields around 100 debug events per second, plenty to diagnose an incident. Errors and warnings are never sampled. Sampling is for high-volume, low-signal events; the correct rate is a function of your traffic, not a universal constant.

New patterns needed:

  • PgBouncer connection pooling plus read replicas
  • Timeout budgets: deadlines propagated down the chain, retries capped and jittered
  • Queue-backed writes to absorb bursts instead of synchronous fan-out
  • Rate limiting at the edge, before requests reach application code
  • Key-splitting for cache hot keys — one tenant's session key accounted for a third of the operations on a single Redis shard

The insight:

At each order of magnitude you are solving different problems. The architecture that works at 1K will not survive 10K, and the 10K design will meet new failure modes at 100K. The only affordable way to find yours early is load testing at 10x before real traffic does it for you — the kind of drill we build into delivery pipelines and release engineering as a standing practice, not a one-off.

Lesson: Scale testing isn't optional. Your 10x traffic day will find every weakness — the only question is whether you find it first.

← Back to Lessons Learned
Tags: #Scaling #Architecture #Complexity #Growth