Severity reflects impact, not cause
A severity level (SEV1, SEV2, ...) is assigned from customer and business impact, before you know why something broke. It sets urgency, staffing, and communication cadence.
Lesson 4 of 10
RCA
Closing the ticket isn't the finish line. A senior TSE turns an incident into a document that rebuilds trust and makes the next one less likely.
What a good RCA actually contains
A root cause analysis is a structured account of what happened, why, and what changes as a result — not a blame report.
Root cause vs. contributing factor
The root cause is the condition that, if removed, would have prevented the incident. Contributing factors made it worse or more likely, but wouldn’t have caused it alone.
A timeline, not just a summary
A minute-by-minute timeline exposes the detection gap (time to notice) and the response gap (time to engage the right people) — details a one-paragraph explanation hides.
Blameless by design
A blameless postmortem asks "what let this happen," not "who caused this." Blame discourages honest reporting and hides the systemic gaps worth fixing.
Specific, owned corrective actions
A weak action item says "add more monitoring." A strong one names the exact signal, an owner, and a target date — otherwise it tends to never get done.
Exercises
Look at the sanitized incident timeline below, then work through both exercises.
14:02 UTC Deploy of v2.14.0 to messaging-api begins
14:06 UTC Error rate on POST /v1/messages climbs from 0.1% to 9%
14:19 UTC Customer opens a support ticket reporting failed sends
14:31 UTC On-call engineer pages in, confirms correlation with the deploy
14:34 UTC Rollback to v2.13.2 initiated
14:39 UTC Error rate returns to baseline
Root cause: v2.14.0 introduced a stricter payload validator that rejected a
previously-accepted phone number format still used by several customers.
Contributing factor: the validator change had no integration test coverage,
and the rollout had no staged or canary phase.Concept check
Answer in your head first, then click a question to reveal the answer.
Scenario checklist
Work through the sample incident timeline and check each item off. Progress is saved in your browser.
Go deeper
These primers are a starting point. Read the primary sources to build real fluency.
Google SRE: Postmortem CultureWhy blameless postmortems build trust and actually reduce recurrenceGoogle SRE: Managing IncidentsRoles, communication, and severity during an active incidentPagerDuty: Incident Response DocumentationAn open framework for severity levels, roles, and postmortem templatesAtlassian: How to run a blameless postmortemA practical walkthrough of the postmortem meeting itself
Training Notes
What building this project taught me.
RCA is a communication skill wearing an engineering hat
The technical content — timelines, root cause, contributing factors — matters, but the lesson keeps circling back to who reads an RCA and what they need from it: leadership needs impact and trust restored, engineering needs a specific fix, and the next on-call needs a faster detection signal.
Fourth repetition of the same exercise shapes, on purpose
Concept check and scenario checklist are now proven across HTTP, Auth, and Events. Repeating the shape for RCA — rather than reaching for a new interaction pattern — keeps the learner’s attention on the content, not on relearning how the page works.