Google gave its outages ranks and commanders, borrowed from the fire line

the problem

Every alert screamed equally loud

background

what everyone would do

Hire more operators to keep pace with growing server count — the traditional operations staffing model, and the one Treynor judged neither affordable nor findable at Google's scale. Even with more people, the underlying problem would remain unsolved: every alert was still treated with equal urgency and every incident handled ad hoc, so more untrained hands improvising a response doesn't fix a coordination problem, it just adds more people improvising.

what they saw

Treynor's team saw that outage response wasn't fundamentally a staffing problem, it was a structure problem — software operations had no equivalent of a fire department's rank system, so every incident competed for the same undifferentiated attention whether it was a minor blip or a company-wide failure. The insight was recognizing that emergency services had already solved exactly this coordination problem, sorting incidents by severity and assigning clear command roles, and that structure could be imported wholesale rather than invented from scratch.

the move

Site Reliability Engineering formalised severity levels, paged tiers and an incident command role copied consciously from emergency services.

why it works

Without a graded severity system, every alert triggers the same response intensity, wasting scarce attention on low-impact problems while potentially under-resourcing genuinely severe ones. Borrowing the fire service's Incident Command System — severity tiers plus a designated Incident Commander, Communications Lead and Operations Lead for each declared incident — creates a standard, pre-agreed structure for how many people respond and who does what, calibrated to the actual scale of the problem. Because responders sort incidents into a known severity tier immediately, response effort scales with actual impact rather than with who happened to notice or how loudly an alert fired, and pre-assigned roles mean a major incident gets an organized, clearly divided response instead of many people independently trying to fix the same thing with no coordination — converting operations from a staffing-intensive model into a structured, scalable one that lets a fixed team handle far more infrastructure without proportional headcount growth.

the payoff

SEV levels and incident command are now default practice across the software industry; Google's incident framework (IMAG) is explicitly built on the fire service's Incident Command System, with a designated Incident Commander, Communications Lead and Operations Lead for every declared incident.

where it breaks

The approach only works when incidents can genuinely be classified into a small number of discrete severity tiers with pre-defined roles — a domain where every incident is truly unique and doesn't fit a graded scheme gets less benefit, since forcing atypical incidents into predefined categories can misallocate the wrong level of response. It also depends on the organization actually training and drilling people in their assigned roles before a real incident happens; a structure that exists on paper but hasn't been practiced under pressure fails the same way an untrained fire crew would, since the value comes from people knowing their role instantly, not from the org chart existing. And it requires genuine discipline in honestly classifying severity — if people inflate severity to get faster attention or downplay it to avoid escalation overhead, the calibration the entire system depends on breaks down.

what came after

Google published its internal practice as the SRE book in 2016, and the severity-tier plus incident-commander pattern it documented — explicitly modeled on FEMA's Incident Command System, created by California firefighters in 1968 — became the default incident-response structure taught and adopted across the software industry, alongside the associated SLO/error-budget framework.

references

1. [1]Incident Response documentation, SRE WorkbookGoogle (official SRE documentation), 2018sre.google
2. [2]The Origins of SRE from the Director of SRE Education at GoogleDevOps Institute, 2021devopsinstitute.com

keep it

same kind of clever

• 1972 · California fire servicesCalifornia's fire agencies built one command system after burning separately
• 1797 · Napoleon's Grande ArméeNapoleon's surgeon treated the worst wounds first, whatever the rank
• 2004 · Craig Fugate / Florida Division of Emergency Management (later FEMA)FEMA gauges a hurricane's damage by whether the local Waffle House kitchen is still serving