The Fix Is One Line in /etc. Just SSH In and Edit It?
Scenario
Production is degraded on a three-node fleet because a kernel tunable in /etc/sysctl.d/ is set to the wrong value. The fix is genuinely trivial: change one line, reload, done. The catch is that this fleet is managed by Ansible — the playbooks are the source of truth, they run on a schedule and on demand, nodes get replaced during deploys, and autoscaling can add fresh instances at any time. A teammate wants to SSH into the three boxes, edit the file by hand, and "codify it in Ansible later, once the fire's out."
The Quick Fix on the Table
Hand-edit the file on all three nodes over SSH right now. It takes five minutes, requires no pull request, and production recovers immediately. The Ansible change gets promised for "later."
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- The scheduled Ansible run un-fixes you. The playbooks own that file. The next scheduled or on-demand run converges it back to the wrong coded value — possibly at 3 a.m. — and the incident reopens with everyone convinced it was already fixed.
- New nodes are born broken. Every autoscaled instance and every node replaced during a deploy is built from the playbooks, which still contain the bad value. The fleet becomes a mix of patched and unpatched machines, and the bug returns intermittently, which is far harder to diagnose than a consistent failure.
- "Later" has the survival rate of a New Year's resolution. Once production is green, the codification task loses all urgency, gets buried under the next fire, and the only record of the fix is in one engineer's shell history.
- Silent drift corrodes trust in the whole system. The moment the team learns that reality and the playbooks can disagree, every future incident starts with "well, is Ansible even accurate?" — and the value of having a source of truth evaporates.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
The answer is not "no hotfixes." Production is degraded; speed matters. The answer is a hotfix with a leash: the manual change and the code change are one unit of work, and the incident is not over until both have landed.
- Declare the emergency change out loud. Post in the incident channel: which file, which value, which hosts, who is doing it. Thirty seconds of typing turns a stealth edit into an auditable action.
- Apply the hotfix — and pause convergence. Make the edit and load it (
sysctl --systemorsysctl -wplus the file change). Then disable the scheduled Ansible run for this play or host group, so the automation can't stampede over the fix while the code catches up. - Codify immediately, not "later." While the incident is still open, change the template or variable in the playbook, open the PR, get an expedited review. This is a 15-minute task when the context is fresh and a forgotten one by tomorrow.
- Re-run Ansible as the verification step. Merge, then run the play against the fleet:
ansible-playbook site.yml --limit affected_group --check --diff, then for real. The correct end state is a run that reports the hand-edited hosts as unchanged — proving code and reality agree — and re-enable the schedule. - Confirm the birth path. Ensure whatever provisions new nodes (the same playbooks, an AMI bake, cloud-init) carries the fix, so the next autoscaling event doesn't resurrect the bug. Spot-check by launching or replacing one node if practical.
- Write the rule down. Add a short emergency-change policy to the runbook: manual changes on managed hosts are permitted during incidents, must be announced, and must be merged into config management before the incident ticket closes. Consider a periodic
--check --diffdrift report so any future silent edits surface within a day.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "I'm not blocking the hotfix — edit the file, fix production now. I'm blocking the 'later': Ansible reverts that file on its next scheduled run, so 'later' means 'until the automation undoes it.'"
- "Every node autoscaling adds tonight is built from the playbooks, which still have the wrong value. Without the code change we get a half-patched fleet and an intermittent bug that's much worse to chase than this one."
- "The codification is one variable and a 15-minute PR while we all have the context. If we defer it, the only documentation of this fix is someone's bash history."
- "Re-running Ansible after the merge is also our verification: if it reports no changes on the hosts we touched, we've proven code and reality match again."
- "The rule I want is simple: SSH fixes during incidents are fine, but the incident isn't closed until the fix is in the repo. That keeps us fast and keeps the source of truth true."