Tests Pass Locally, Fail in CI. Just Retry?
Scenario
The test suite is green on every developer laptop but fails intermittently in CI. A look under the hood shows the usual suspects: dependency versions are not pinned, the CI runners use a different operating system than the laptops, several tests depend on the current date and time, and a few share temporary directories. Re-running a failed job often turns it green — which is exactly what people have started doing.
The Quick Fix on the Table
Make the workaround official: configure the pipeline to automatically retry failed jobs until they pass. The build goes green, releases stop getting blocked, and nobody has to dig through test internals.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- Retries erase the signal, not the problem. An intermittent failure is information: something in the code or tests behaves non-deterministically. Auto-retry configures the pipeline to discard that information at the exact moment it appears.
- Some of those failures are real bugs. Race conditions, timezone handling and shared-state collisions fail intermittently in tests because they fail intermittently in production. Retrying until green ships them.
- Trust in CI collapses. Once red might mean "flaky", every failure gets a shrug and a re-run — including the genuine regression. The suite's entire purpose, a trustworthy verdict, is gone.
- The suite gets slower and flakier. Retries multiply CI time and cost, and with the pressure removed, new flaky tests accumulate unchecked. Teams that go down this road end up with triple-retry pipelines that take an hour to lie to them.
- The known causes stay unfixed. Unpinned dependencies and OS mismatch don't just flake tests — they mean production dependencies drift too. Retrying papers over a reproducibility problem much bigger than CI.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Treat each flaky test as a defect with an owner. Quarantine identified flakes into a separate, non-blocking job and file a ticket per test — visible, tracked and temporary, unlike a silent global retry.
- Pin dependencies everywhere. Commit lockfiles (
package-lock.json,poetry.lock,go.sum) and install from them in CI (npm cirather thannpm install) so laptops and runners resolve identical versions. - Close the environment gap. Run the suite in a container image shared by CI and local development, so "different OS" stops being a variable —
docker compose run testsbehaves the same on both. - Remove real-time dependencies from tests. Inject a fixed clock or freeze time in test setup instead of calling "now" — tests that depend on the wall clock fail at midnight, month ends and DST changes by design.
- Isolate shared state. Give each test a unique temp directory (
mktemp -dor the framework's tmp-path fixture) and unique database schemas/fixtures, so parallel tests stop trampling each other. - Reproduce the CI conditions to flush out the rest. Run the suite locally with the CI's parallelism and a randomized test order — most "CI-only" failures reproduce immediately once ordering and concurrency match.
- Keep score. Track pass rates per test in CI so new flakes surface within days; a small dashboard beats folklore about "that one test".
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "A test that fails one run in five is telling us something is non-deterministic. Auto-retry doesn't fix that — it just deletes the message."
- "Race conditions and time bugs flake in CI because they'll flake in production. The retry button ships them to customers."
- "Once we retry by default, a red build stops meaning anything — and the day we ignore a real regression because 'it's probably flaky' costs more than every flake combined."
- "We already know the causes: unpinned dependencies, OS mismatch, clock and temp-dir coupling. Each is a bounded fix, not an open-ended hunt."
- "I'm fine unblocking today's release by quarantining the known flakes — visibly, with tickets. That keeps the pressure to fix them; a global retry removes it forever."