We scaled horizontally to 50 instances. The bottleneck just moved.

Phase 1: Application bottleneck

  • 5 app servers, 100% CPU
  • Solution: Scale to 20 servers
  • Result: App CPU 30%

Phase 2: Database bottleneck

  • Database: 100% CPU, 5000 connections
  • Solution: Connection pooling, read replicas
  • Result: Database CPU 60%

Phase 3: Load balancer bottleneck

  • ALB scaling lagged behind a sharp traffic spike — ALBs scale automatically on LCUs, but not instantly, and our spike outran it
  • Solution: pre-scaled ahead of known traffic events (coordinated with AWS support) and moved the spikiest internal TCP traffic to an NLB
  • Result: Traffic distributed

Phase 4: Message queue bottleneck

  • RabbitMQ single node saturated
  • Solution: Clustered queue, partitioned topics
  • Result: Messages flowing

The lesson:

Theory of constraints, applied to systems:

System throughput = min(
    app_capacity,
    db_capacity,
    network_capacity,
    queue_capacity,
    external_api_capacity
)

What we learned:

  • Identify the bottleneck BEFORE scaling
  • Load test the entire path
  • Monitor all components, not just apps
  • Sometimes vertical scaling is cheaper

Full-path load testing — client to database and back, not just the app tier — is a standard exercise in our DevOps engagements, because the component you didn't test is where the next bottleneck lives.

Lesson: Scaling one component moves the bottleneck to the next. Plan for end-to-end capacity.

← Back to Lessons Learned
Tags: #Scaling #Performance #Bottleneck #Architecture