Incidents are not exceptional events; they are a normal output of running systems. What separates teams is not whether things break but how predictably they respond when things do. A working incident process fits on one page: clear severity levels, named roles, runbooks people can follow under stress, alerts tied to what users experience, and a postmortem habit that turns each incident into a system improvement. This playbook covers each piece.

Severity Classification

Severity exists to answer one question fast: how hard do we mobilize? Three levels are enough for most organizations. SEV1: users cannot use the product, data is at risk, or a security breach is in progress — all hands, executive notification, public status page. SEV2: significant degradation with a workaround, or a major feature down — a dedicated responder team, stakeholder updates on a fixed cadence. SEV3: minor impact or elevated risk — handled by the owning team in working hours.

Two rules make the scheme work. Anyone can declare an incident — if declaring requires a manager's blessing, you have added latency exactly where it hurts most. And when in doubt, classify high and downgrade later; the reverse pattern, upgrading an incident an hour in, is how minor outages become long ones.

Roles: The Incident Commander Does Not Debug

The single most valuable structural change most teams can make is separating coordination from investigation. The incident commander (IC) owns the response: tracks state, assigns work, decides on mitigations, and keeps the incident channel from becoming forty people asking "any update?". The IC does not touch a terminal. The communications lead owns the outward flow — status page updates, stakeholder messages, the promise of the next update and its delivery. The operations lead drives the technical investigation with whoever else is needed.

On a small team one person may wear all three hats for a SEV3 — that is fine, as long as the hats exist and a SEV1 splits them. The failure mode to design against is the best engineer simultaneously debugging, coordinating and fielding executive questions, doing all three badly.

Runbooks That Work at 3 a.m.

A runbook is judged in the worst conditions: a paged engineer, half-awake, seeing this alert for the first time. That sets the quality bar. Every page-level alert links directly to its runbook. The first section is diagnostic — three commands that establish what is actually happening — before any remediation. Commands are copy-pastable, not described in prose. The escalation path names a team and a channel, not a person who left last year. And runbooks are exercised: a game day that walks the on-call rotation through the top five runbooks finds the stale ones before an outage does.

SLO-Driven Alerting

Paging on causes — CPU high, disk at 80%, a pod restarting — produces noise, and noise produces ignored pages. Page on symptoms instead: the service level objective is being burned. If you promise 99.9% availability, your error budget is 0.1% of requests; a page should fire when that budget is being consumed fast enough to matter. In Prometheus terms, a fast-burn rule for a 99.9% SLO looks like this:

- alert: HighErrorBudgetBurn
  expr: (
      sum(rate(http_requests_total{code=~"5.."}[5m]))
    /
      sum(rate(http_requests_total[5m]))
  ) > (14.4 * 0.001)
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "Fast burn: error rate is 14.4x the budget for a 99.9% SLO"

Pair a fast-burn rule (short window, high multiplier, pages) with a slow-burn rule (long window, low multiplier, opens a ticket). Cause-based signals still exist — as dashboards and tickets, not pages. The practical effect: every page means users are affected, so every page gets taken seriously. Security-relevant events deserve the same discipline; our overview of SIEM systems covers how detection and alerting converge on that side.

Blameless Postmortems

The postmortem is where an incident becomes an investment. Blameless does not mean consequence-free; it means the analysis assumes people acted reasonably on the information they had, and asks why the system made the wrong action easy or the right one hard. A useful postmortem contains a precise timeline, the full set of contributing factors — resist the fiction of a single root cause — and action items that each have an owner and a due date. An action item without an owner is a wish. Review open items in a standing forum, and share postmortems widely: an outage the whole organization learns from was not entirely wasted.

None of this requires new tooling — it requires deciding the process in calm conditions rather than inventing it during an outage. If you want help standing up severity schemes, on-call structure and SLO-based alerting — or an incident process that also covers security events — our SecOps solutions team does this work, and our managed services practice can carry the on-call load with you.

Tags: #IncidentResponse #SRE #SLO #OnCall #Postmortem