The Pipeline Is Slow. Skip Tests to Ship?
Scenario
The pipeline takes about 40 minutes end to end, and nearly all of it is the test stage: unit, integration, and end-to-end suites running one after another on a single runner, with no caching between runs. Every merge sits in the queue, hotfixes crawl, and the team's release rhythm is set by the slowest job in CI. Frustration peaks, and a proposal lands: pull the test stage out of the deploy path entirely. Developers can run tests locally before pushing, and releases become instant.
The Quick Fix on the Table
Delete (or make optional) the test stage. Shipping goes from 40 minutes to a couple of minutes today, with zero engineering effort. Quality is delegated to "everyone runs the tests locally, promise."
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- "Run it locally" is a policy, not a control. Under the same deadline pressure that produced this proposal, local test runs will be skipped — silently, and most often on exactly the risky changes. CI's whole value is that it cannot be forgotten.
- Local machines don't test integration. Developers run a subset against their own environment and their own branch. Only CI tests the merged result against a clean, production-like environment — the place where "works on my machine" goes to die.
- The cost doesn't disappear; it moves to production. A regression caught by CI costs 40 minutes. The same regression caught by customers costs an incident, a rollback, a war room, and trust — a terrible trade against the minutes saved.
- Speed actually gets worse over time. Once trunk can be broken by any push, every deploy becomes a gamble and every incident triggers "who didn't run the tests?" archaeology. Teams in that state ship less often, more fearfully — the opposite of the goal.
- It's solving the wrong variable. Nobody has asked why the suites take 40 minutes. Sequential execution, one runner, no caching — the pipeline is slow because it was never engineered, not because testing is inherently slow.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Profile before touching anything. Break the 40 minutes down per stage and per suite from the CI logs. Typically a handful of E2E tests and repeated dependency installs dominate; you can't optimize what you haven't measured.
- Run the suites in parallel, not in sequence. Unit, integration, and E2E don't depend on each other's results — fan them out as concurrent jobs. On most CI systems this is a YAML restructure, and the wall-clock time drops to the longest single suite immediately.
- Shard the big suites. Split tests across multiple runners by timing data (
pytest-xdist,jest --shard, or your CI's native test splitting). Four shards of a 20-minute suite is a 5-minute suite. - Cache the setup. Cache dependency directories and Docker layers between runs (keyed on lockfile hashes), and prebuild a test base image so jobs start testing in seconds instead of reinstalling the world every run.
- Stage the feedback. Fast checks (lint, unit) gate the PR in a couple of minutes; integration and E2E run in parallel before merge or on merge to main. Developers get their tight feedback loop without any test being deleted from the path to production.
- Prune and stabilize. Delete redundant tests, push slow E2E scenarios down the pyramid into integration or unit tests where possible, and quarantine flaky tests with an owner and a deadline — flakes that cause reruns are often a huge hidden chunk of "pipeline time."
- Set a budget and defend it. Agree on a target (e.g. PR feedback under 10 minutes), put pipeline duration on a dashboard, and treat regressions of the budget like performance bugs. That keeps this fix from decaying back to 40 minutes.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "The proposal replaces an automatic guarantee with a promise, and the promise breaks under exactly the deadline pressure that made us want to skip tests in the first place."
- "The pipeline is slow because everything runs sequentially on one runner with no caching. That's an engineering problem with a known fix — parallelize, shard, cache — and it likely gets us under 10 minutes without removing a single test."
- "A test failure in CI costs us 40 minutes. The same bug in production costs an incident and customer trust. We'd be trading our cheapest safety net for our most expensive failure mode."
- "Local runs can't test the merged code against a clean environment — that integration gap is precisely where the ugly bugs live, and only CI covers it."
- "Give me one sprint to restructure the pipeline. If PR feedback isn't dramatically faster, we revisit — but let's make the tests fast before we make them optional."