Deploys Are SSH + git pull at 5pm Friday. Why Change It?
Scenario
Production is a monolith on two VMs behind a load balancer. Releasing means SSHing into each box, running git pull, running npm install if somebody remembers that dependencies changed, and restarting the service. CI exists, but only to check pull requests — it never touches production. Last month one VM had local edits that made the pull half-apply, and the two servers quietly ran different code for a day. The next release is scheduled for 5pm on Friday.
The Quick Fix on the Table
Technically the proposal is to change nothing: "We've deployed this way for two years and it has always worked — why add ceremony?" The status quo is fast, everyone knows the steps, and a pipeline sounds like process for its own sake.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- "It has always worked" is survivorship bias — and it already didn't. The mismatched-VMs incident last month is exactly the failure class this method produces; it just happened to be cheap that time.
- The process depends on human memory. "Run npm install if someone remembers" means every deploy is a dice roll on whether dependencies, migrations and restarts all happen, in the right order, on every box.
- There is no rollback. When 5pm Friday goes wrong, the recovery plan is a human reverse-engineering git commands on two servers under pressure — into the weekend, with nobody around.
- Servers drift apart silently. Local edits, partial pulls and skipped installs mean the two VMs are only probably identical. Behind a load balancer, that turns into bugs that appear for 50% of requests and vanish on refresh.
- Building on the production box is backwards. Dependency installs and builds running on live servers consume production CPU, can fail halfway, and mean what you tested in CI is not the artifact you run in production.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Make CI produce a deployable artifact. The pipeline that already validates PRs should also build the release — a versioned tarball or container image with dependencies baked in — so production never runs
npm installagain. - Script the deploy end to end. One entry point (a deploy script,
ansible-playbook, or a CI deploy job) that fetches the artifact, switches acurrentsymlink to the new release directory, and runssystemctl restart app— identically on every VM, no memory required. - Keep previous releases for instant rollback. With releases in versioned directories, rollback is repointing the symlink and restarting — seconds, not an archaeology session over SSH.
- Deploy through the load balancer, one node at a time. Drain a VM, deploy, run a health check with
curlagainst the app, re-enable it, then do the next — a bad release stops at the first node instead of taking down both. - Make production read-only for humans. No local edits on servers; if a hotfix is needed it goes through the same pipeline. This kills the drift that caused last month's incident.
- Set a deploy window policy. Ship when the team is around to watch and respond — which specifically means not 5pm Friday. With fast rollback in place this becomes a guideline rather than fear, but the default should still respect coverage.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "Two years of this working is luck plus low traffic, not a safety property — and last month's mismatched servers is what the failure looks like when the luck runs out."
- "I'm not proposing ceremony. I'm proposing that the deploy be one command that works the same every time, instead of a checklist that lives in people's heads."
- "The question that decides this: if Friday's 5pm deploy breaks production, what exactly is our rollback? Today the honest answer is 'someone SSHs in and improvises all weekend'."
- "CI already builds and tests the code — extending it to produce the artifact we actually deploy is a small step, and it means production runs exactly what we tested."
- "The migration can be incremental: script the current steps first, then artifacts, then rolling deploys. Each step pays for itself; nothing requires a big-bang rewrite."