Career & Craft
Incident Response & Postmortems
The practice of handling production failure: detect, communicate, mitigate first — roll back, flip the flag — and diagnose fully later, then write a blameless postmortem so the same failure cannot recur silently. Blameless matters because fear hides information. The real output is action items with owners; a postmortem without follow-through is a eulogy.
Purpose
Incidents are inevitable; incident response is what turns them from chaos into procedure. During the fire, the priorities invert from normal engineering: restore service first, understand later — a rollback that fixes the symptom beats a root cause found at hour three. Afterwards, the postmortem converts the outage into its only durable value: knowledge of how the system actually fails, and changes that remove that path.
When to Use It
Any user-facing failure: severity levels decide who wakes up and how loudly to communicate; an incident commander role keeps one person coordinating instead of five people debugging in parallel silence; status updates on a cadence keep stakeholders from becoming a second incident. Postmortems apply beyond outages — near-misses and painful deploys teach the same lessons at lower tuition.
Trade-offs
Blameless is the load-bearing word: name a culprit and the next incident's timeline will be quietly incomplete — you trade the comfort of accountability-theatre for accurate information, which is the better deal every time. The failure mode of the ritual is documents that read well and change nothing: action items with no owner, no deadline and no tracking. Also resist the "human error" conclusion; a system one mistake away from an outage is the actual finding.
Implementation
Define severities and an on-call rotation before you need them; templates and runbooks beat improvisation at 3 a.m. During: mitigate (rollback, feature flag off, failover), keep a timestamped log of actions and observations — it becomes the postmortem's spine. After: write timeline, impact, contributing factors (plural — ask "why" past the first answer), what went well, and tracked action items with owners; review them in the same ritual as new work.