p99 Latency Spiked Across a Microservice Chain
Scenario
A user-facing request fans through a chain of six microservices before the response goes back out. This week the p99 latency of that endpoint jumped sharply, while the average barely moved — most users are fine, but the slowest one percent are having a terrible time, and those are often your heaviest users. You have per-service CPU and memory dashboards, request logs, and a distributed tracing system that was set up a while ago and is rarely opened. On the dashboards, one service in the chain runs visibly hotter on CPU than its neighbors.
In triage, a colleague draws the obvious line between the two facts: "Service C is the one with high CPU — scale it up and the p99 will come back down." It is a five-minute change to a replica count, and it feels like acting on data.
The Quick Fix on the Table
Bump the replica count (or instance size) of the service with the highest CPU. No code changes, instant to apply, and there is a dashboard screenshot to justify it.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- High CPU is a correlation, not a diagnosis. The hottest service may simply be the one doing legitimate compute-heavy work, efficiently. Meanwhile the p99 culprit is often a quiet service waiting on a lock, a saturated connection pool, a slow downstream query, or GC pauses — none of which show up as sustained CPU.
- Averages and per-service views can't see tail behavior. A p99 spike at the edge with flat averages is the classic signature of tail latency amplification: with six sequential hops, a request only needs one hop to hit that hop's slow tail. Even if each service's p99 is rare, the chain multiplies the chance of hitting at least one — per-service dashboards each look "mostly fine" while the composite tail explodes.
- Queueing effects hide from resource metrics. Time spent waiting in a queue, in a connection pool, or on a retry backoff is latency the user feels but no CPU/memory graph records. Scaling CPU does nothing for wait states.
- A wrong scale-up costs twice. You pay for the extra replicas, and — worse — the team believes the problem is addressed. When p99 stays high, you have burned time, money, and credibility, and you are back at square one with less of each.
- It can even make things worse. If the real bottleneck is a shared database or a downstream rate limit, adding replicas upstream increases concurrent pressure on it, deepening the tail.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Start from slow traces, not hot dashboards. In your tracing backend (Jaeger, Tempo, X-Ray, Honeycomb — whatever is wired up), filter traces for the affected endpoint with
duration > p99over the spike window. The question is not "which service is busy" but "where do the slow requests spend their time." - Read the span waterfall for the dominant gap. In a slow trace, look for the longest span — and just as important, for unexplained gaps between spans (time in queues, connection acquisition, retries) which often exceed any instrumented span. Compare against a fast trace of the same route to see what differs.
- Compare a sample, not one anecdote. Pull 20–50 slow traces and tally where the time goes. If 80% of them stall in the same hop or the same database call, you have the bottleneck with evidence; if the slowness is spread evenly, you are looking at systemic amplification (retries, timeouts, fan-out) rather than one bad service.
- Drill into the guilty hop with targeted signals. Once tracing points at a service, use its specific telemetry: connection pool saturation, GC pause metrics, lock waits, slow query logs (
pg_stat_statements, RDS Performance Insights), thread pool queue depth. Exemplars — trace IDs attached to latency histogram buckets — let you jump from a Prometheus spike straight to a guilty trace. - Check the tail-amplifiers in the chain itself. Audit timeout and retry configuration hop by hop: retries without budgets, timeouts longer than the caller's, and sequential calls that could be parallel all convert one slow dependency into an edge-wide p99 spike.
- Fix the identified cause, then verify on the same traces. Whether the fix is an index, a pool size, a retry budget, or — legitimately — scaling one service, re-run the trace query afterwards. p99 back to baseline in the trace data is the acceptance test, not a calmer CPU graph.
- Make the tracing habit permanent. Ensure context propagation covers all six hops, keep tail-based sampling so slow traces are always retained, and put an exemplar-linked latency panel on the endpoint's dashboard so the next spike starts at step 1 automatically.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "High CPU tells us that service is working hard, not that it's making requests slow. The p99 culprit is usually waiting — on locks, pools, or a downstream — and waiting doesn't show up on a CPU graph."
- "With six hops in a chain, a request only needs one of them to hit a slow tail. Every per-service dashboard can look healthy while the end-to-end p99 burns — that's exactly why we have tracing."
- "Thirty minutes in the tracing UI gives us the answer with evidence: where slow requests actually spend their time. A scale-up is a guess we pay for whether or not it's right."
- "If the traces say Service C is the bottleneck, I'll scale it immediately — the same data that stops a wrong fix fast-tracks the right one."
- "The follow-up matters too: retry and timeout settings across the chain decide whether one slow dependency stays local or amplifies to the edge. That audit is free and permanent."