Service A Intermittently Cannot Reach Service B
Scenario
Inside your Kubernetes cluster, Service A calls Service B and roughly one call in twenty times out. Both are ordinary Deployments with several replicas behind ClusterIP Services, CoreDNS handles discovery, and NetworkPolicies are in place. Nineteen calls out of twenty are perfectly healthy, application logs show no pattern — no particular pod, no particular payload, no particular time of day — and nothing is obviously broken. That is exactly what makes it maddening.
After a couple of inconclusive log-reading sessions, a teammate proposes closure: "It's five percent. Add retries to the client and move on — we have real work to do."
The Quick Fix on the Table
Wrap the call in a retry policy and declare the ticket done. It genuinely makes the user-visible symptom vanish, it is a ten-line change, and nobody has to descend into the networking stack.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Retries mask the signal, they don't remove the fault. The 5% failure rate is a symptom of something specific — and that something (DNS, conntrack, endpoint churn, a policy race) is still there, free to grow. When it hits 30% during the next traffic spike or rollout, you'll be debugging it blind, because retries have been hiding the trend for months.
- Retries without diagnosis can be actively dangerous. If the timeouts come from overload or connection-table exhaustion, retries multiply traffic exactly when the system is weakest. And if any hop of the call is not idempotent, retried requests mean duplicate side effects.
- The latency cost is real and permanent. A retry after a multi-second timeout turns a 50ms call into a multi-second one for 5% of requests — you have converted hard failures into a fat p99 tail and called it a fix.
- Intermittent in-cluster failures have a short, known suspect list. This class of bug is not mysterious; it is a well-mapped set of Kubernetes failure modes that a systematic hour of diagnosis can usually pin down. Skipping that hour is a choice, not a necessity.
- "5% is acceptable" quietly becomes the culture. The next flaky thing gets a retry too, then the next. Eventually the cluster's baseline is a stack of masked faults nobody understands, and every real incident starts from that fog.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- First, characterize the failure precisely. Is the timeout on DNS resolution, TCP connect, or response read? Instrument the client (or capture with
tcpdumpon a sidecar/node) to split those phases. Each points at a different subsystem — this one distinction cuts the suspect list in half. - Check the DNS path and the ndots trap. With the default
ndots:5, a lookup forservice-btries multiple search-domain suffixes first, multiplying query load on CoreDNS, and UDP responses can be silently dropped. Look for errors in CoreDNS logs and itscoredns_dns_request_duration_secondsmetrics; mitigate with FQDNs (service-b.namespace.svc.cluster.local.), a tuneddnsConfig, or NodeLocal DNSCache. - Test for the conntrack/DNAT race. Under kube-proxy iptables mode, there is a known race inserting conntrack entries for UDP (and connection reuse patterns) that manifests as exactly this kind of low-rate timeout. Check
conntrack -Sforinsert_failedcounters on the nodes, and verify the conntrack table isn't near its max (net.netfilter.nf_conntrack_max). - Correlate failures with endpoint churn. Overlay the timeout timestamps with Service B's rollout and scaling events (
kubectl get events, deployment history). Pods that receive traffic before readiness, or terminate without a grace period draining connections, produce failures precisely during churn. Fix with proper readiness probes, apreStopsleep, and graceful shutdown handling in Service B. - Audit NetworkPolicies as a set, not a document. Confirm the actual behavior: from an A pod, loop
curlagainst each individual B pod IP (bypassing the Service) and against the ClusterIP. Per-pod failures implicate policy or CNI on specific nodes; ClusterIP-only failures implicate kube-proxy/DNS. Check the CNI agent logs on the nodes hosting failing pods. - Fix the identified cause, and verify with numbers. Whatever the culprit, the acceptance test is the failure rate itself: measure timeouts per thousand calls before and after. "It feels better" doesn't close this ticket; 0.0% over a week does.
- Then add retries — as engineering, not as a shrug. A capped, jittered retry with a tight per-try timeout and a retry budget is legitimate defense-in-depth after the root cause is fixed, sized against a known failure profile, with metrics on retry rate so a regression is visible instead of masked.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Retries make the symptom invisible, not the fault gone. Whatever kills one call in twenty is still in the cluster, and the day it gets worse we'll have no data trail because we spent months hiding it."
- "This failure signature — intermittent, no app-log pattern, in-cluster — has a short known suspect list: DNS and ndots, the conntrack race, endpoint churn during rollouts, or a policy/CNI issue. One structured afternoon can usually name the culprit."
- "If the cause is overload or connection-table pressure, retries pour fuel on it — we'd be tripling traffic exactly when the system is struggling. And if any of these calls isn't idempotent, retries mean duplicated side effects."
- "A retry after a timeout turns a 50-millisecond call into a multi-second one. That's not fixing the 5% — that's moving it into our p99 and paying for it forever."
- "I'm not anti-retry — I want retries with a budget and a per-try timeout as a safety net once we know what they're covering for. Diagnose first, then add the armor."