We adopted Istio mainly for observability. Requests got visibly slower, we measured about 15ms of added latency per hop, and we ripped the mesh out. Half of that decision was right. The measurement that drove it was not.

What we measured versus what it meant

A properly resourced sidecar adds low single-digit milliseconds per hop — Istio's own published benchmarks have shown that for years. Fifteen milliseconds per hop is not the price of a service mesh; it is the symptom of a starved one. Ours was starved: we had copy-pasted proxy CPU limits so low that the sidecars were constantly throttled under load. We benchmarked our misconfiguration and billed it to the technology.

Our architecture amplified it. The checkout path chained a dozen sequential internal calls — an architectural smell in its own right. At a realistic ~2ms per hop that is roughly 25ms of mesh overhead: noticeable, arguable. At our throttled 15ms it was ~180ms on a 200ms latency budget, which is how "remove Istio" became an easy sell.

What we actually needed

  • Distributed tracing: OpenTelemetry SDKs in-process — no proxy required
  • Circuit breakers and retries: library-based (Resilience4j)
  • Traffic management: plain Kubernetes Services were sufficient at our scale
  • mTLS: the one genuine mesh capability on our list. We deferred it deliberately, with a documented decision and a date to revisit — not because encrypting service-to-service traffic "wasn't in our threat model." Zero-trust for internal traffic is table stakes in 2026; deferrals are acceptable, permanent dismissals are not.

The 2026 landscape we initially ignored

The per-hop sidecar tax is no longer the only deal on the table. Istio's ambient mode — GA since 2024 — removes sidecars entirely and handles L4 with a shared ztunnel at sub-millisecond cost, and eBPF-based options like Cilium's mesh do similar work in-kernel. If mTLS and traffic policy are what you need, sidecar-less meshes deliver them at a fraction of the classic overhead. Evaluate them before writing off the category, especially once you are running Kubernetes in production at any real size.

Where we landed

  • Removed the sidecar mesh; added OTel tracing and library resilience — latency returned to baseline and cluster resource usage dropped roughly 40%
  • mTLS revisit is scheduled, with ambient mode as the leading candidate

Lesson: Don't adopt a mesh for observability alone — and don't reject one on numbers you haven't traced to their cause. Benchmark a correctly configured deployment, price the overhead honestly, then decide.

← Back to Lessons Learned
Tags: #ServiceMesh #Kubernetes #Performance #ZeroTrust