On-Call Is Drowning in Alerts. Silence Them?
Scenario
A legacy alert pages on-call whenever CPU crosses 80% for a minute. It fires several times most nights. Every time, latency and error rates are normal, users notice nothing, and the responder acknowledges it and goes back to sleep — a little more tired and a little more trained to assume the pager is crying wolf. Nobody remembers who created the rule or what it was supposed to catch. The team is worn down, and the on-call rotation is becoming something people dread.
The Quick Fix on the Table
A justifiably exhausted teammate wants to slap a permanent silence on the alert — or just delete the notification route. Thirty seconds of work, and tonight everyone sleeps.
The quick fix is on the table and the room is waiting for your call. Would you sign off on it? Take a position and justify it — out loud or on paper — before revealing the analysis.
Why the Quick Fix Fails
- A silence is a decision nobody made. Muting the alert without analysis means someone once decided this condition was page-worthy, and now someone else decided it isn't — with no record, no review, and no answer to "what was it for?". Silences also outlive their context: the classic postmortem line is "there was an alert for this, but it had been silenced for months".
- It treats one alert, not the disease. The team is drowning because the alerting system pages on causes (CPU, disk, restarts) instead of symptoms (latency, errors, user impact). Silence this rule and the next noisy one takes its place; the fatigue economics haven't changed.
- Alert fatigue is itself an outage risk. Responders trained by false alarms acknowledge on autopilot. The cost of the noise isn't just lost sleep — it's the added minutes of hesitation on the night a page is real. Silencing one alert doesn't rebuild that trust; a trustworthy pager does.
- The signal might have a valid non-paging use. Sustained high CPU is a legitimate capacity signal. Deleting it entirely throws away trend data someone will want at planning time; the mistake was routing it to a human at 3 AM, not measuring it.
The interviewer nods: “Fine, the quick fix is off the table. So what exactly would you do — step by step?” Sketch your plan before revealing the approach.
The Right Approach
- Adopt one paging principle: a page means a human must act now. If no user-visible impact is happening or imminent, and there is no action to take, it must not page. Write this down as team policy — it is the yardstick every existing and future alert gets measured against.
- Audit the inventory with data. Pull the last 30–90 days from PagerDuty/Opsgenie and label every alert: how often it fired, how often it was actionable, what action was taken. The CPU alert will score "dozens of pages, zero actions" — now its removal is an evidence-based decision, not a tired one.
- Rebuild paging around symptoms and SLOs. Page on what users experience: error rate, latency, availability — ideally as SLO burn-rate alerts (e.g. multi-window burn rates in Prometheus/Alertmanager), which page fast on fast burns and stay quiet on noise. Cause-level signals become diagnostics you consult after a symptom fires, not pagers themselves.
- Introduce severity tiers with different delivery. Page (wake a human) for user impact; ticket (next business day) for things like "disk 80% full, ~2 weeks to full"; dashboard/log only for everything else. The CPU signal moves to a capacity dashboard and maybe a ticket threshold — retired as a pager, retained as a metric.
- Delete non-actionable alerts without guilt — through the process. Every alert that fails the "action now?" test is fixed (better threshold, longer duration, better signal) or removed, with a one-line rationale in version-controlled alert definitions so the next engineer knows why.
- Give every remaining page a runbook. If nobody can write down what the responder should do when it fires, that is the proof it should not page.
- Review the pager like you review costs. A short monthly look at pages-per-on-call-week, actionable percentage, and any alert that fired more than a few times without action. Alert quality decays; the review is the maintenance loop.
Final pushback: “Your plan costs more time and money than the quick fix. Convince me.” How do you defend your position under pressure?
How to Defend It
- "I want that alert gone too — but deleted deliberately with the data in front of us, not muted at 3 AM. A silence is how alerts end up disabled during the incident they were meant to catch."
- "The problem isn't this alert; it's that we page on causes instead of symptoms. Fix only this one and next month a different rule is ruining our sleep."
- "Every false page trains us to ack without looking. The real cost of the noise is the slow reaction on the night the page is real — alert fatigue is an outage risk, not just an annoyance."
- "The bar is simple: a page means a human must act now. High CPU with flat latency and zero errors fails that bar — it belongs on a capacity dashboard, not a pager."
- "Give me one sprint to run the audit and move us to symptom-based paging. The outcome is fewer pages and more trust in each one — strictly better than one silence for both sleep and safety."