Partial Failure Hell
In microservices, everything can partially fail. 1 of 8 services went down. The result was worse than a complete outage.
The scenario:
- User flow: 8 services in sequence
- Service 6 (recommendations): 50% failure rate per attempt
- No circuit breaker, no fallback
- Retry policy: 3 attempts with a 30-second timeout each
What users experienced:
- About half of requests succeeded on the first attempt
- Most of the rest succeeded "eventually" — after sitting through one or two 30-second timeouts
- Roughly 1 in 8 requests burned all three attempts and failed after 90+ seconds
- Users refreshed → more load on an already-failing service
- Threads blocked on those 30-second waits exhausted pools across all upstream services — everything degraded
The worst part:
- Dashboards showed "Service 6: 50% healthy"
- User-facing experience: 0% usable
- No clear owner (whose fault is partial failure?)
The fix:
- Circuit breakers on all outbound calls
- Graceful degradation (show page without recommendations)
- Timeouts measured in ms, not seconds
- Bulkhead pattern to isolate failures
Resilience patterns like these are standard scope in our DevOps and reliability work — retro-fitting them mid-incident is the expensive way to learn them.
Lesson: In distributed systems, design for partial failure first.