Writing an incident review people will read

By Lior Rabanian · · 6 min read
  • Incidents
  • Method
  • Work

The outage ends at 3am. Everyone is relieved, exhausted, and quietly certain they will remember what happened. Nine days later somebody writes a document from Slack scrollback and half-memories, it gets a meeting, it gets filed, and nothing changes.

That is the normal outcome, and the causes are structural rather than a failure of anyone's diligence.

Why most reviews change nothing

Written too late. Memory of an incident degrades unusually fast, because most of it was formed under stress and much of it was wrong at the time. What survives at day nine is a tidy narrative in which the sequence made sense — which is precisely the version that contains no lessons.

The timeline is reconstructed instead of recorded. A timeline assembled afterwards has hindsight in every line. "At 02:14 we realised it was the cache" is nearly always wrong; at 02:14 someone suspected the cache, among three other theories. The suspicion and the alternatives are the interesting part, and reconstruction deletes them.

Blameless is stated but not practised. If the document says blameless and the meeting looks for who deployed it, everyone learns which of the two is real, and the next review will be written defensively. A defensive review has no useful content.

The actions are aspirational. "Improve monitoring" is not an action. It has no owner, no date, and no definition of done, and it will be on the next review too.

Root cause, singular. Complex systems do not have a root cause. They have a set of conditions that were individually survivable, and the search for the one cause stops the investigation at whatever is easiest to blame — usually a person or the last change.

One note holding the timeline, what people believed at the time, and what actually changed afterwards
Recorded as it happens, including the wrong theories. Reconstructed later, the wrong theories are the first thing to disappear.

Record during, not after

The highest-leverage change, and it costs one person's partial attention.

During an incident, someone writes timestamped lines as things happen. Not analysis — observation:

02:04  alerts on checkout latency
02:09  suspect the deploy at 01:50, start rollback
02:14  rollback done, no change — so not the deploy
02:20  noticed cache hit rate fell at 01:30, before the deploy
02:31  restarted the cache nodes, latency recovering
02:46  green. leaving the rollback in place, unclear why 01:30

Six lines, written in the moment, and they contain something no later reconstruction can: the wrong theory, and how long it cost. Twelve minutes went to the deploy hypothesis. That is a finding — it says something about what the dashboards were showing and what people reached for first — and it is invisible in every version of this document written the following week.

The person taking notes should not be the person fixing. It is a real role, it is the easiest one to fill, and it is what makes the difference between a review with content and a review with a narrative.

The five sections

Longer templates exist. These five carry the value.

What people experienced. Not "elevated 5xx" — "customers could not check out for 42 minutes, about 900 attempts failed." Impact in human terms, at the top, because it is the only part most readers will read.

The timeline. As recorded, including the wrong turns, with times. If it is tidy, it has been laundered.

What made this possible. Plural, deliberately. The change that triggered it, the monitoring that did not fire, the alert that fired and was muted last month for good reasons, the runbook that was out of date, the fact that only one person understood the system. Every one of these is a contributing condition, and every one is an opportunity. Naming one of them "the" root cause discards the rest.

What made it hard to fix. Frequently more valuable than the cause. Nobody could find the dashboard, the on-call did not have access, the rollback took nine minutes, the logs were in a format nobody could read at 2am. Time-to-recover is where most of the real damage is, and it is the part most reviews barely mention.

What we are changing. Two or three items, each with a name and a date. Not eleven. Eleven means none, and everyone in the room knows it.

The blameless part, done properly

Not "we will not say who". That is silence, not blamelessness.

It is: assume everyone acted sensibly given what they knew at the time, and ask what made the sensible action wrong. Somebody deployed on a Friday afternoon — why did that seem fine? Somebody muted the alert — what was it doing that made muting reasonable? Those questions produce fixable answers. "Who deployed it" produces a person who will be more careful and a system that is exactly as fragile as before.

The practical test is whether people write down their own mistakes in the timeline. If they do, you have it. If the timeline is written in the passive voice throughout, you do not, whatever the template says.

What happens afterwards

Two things, neither of which is the meeting.

The actions become real dated tasks with owners, in whatever system that team actually uses, that day. An action item living only inside a document is a wish. This is the step that separates teams whose reviews change things from teams whose reviews accumulate.

Somebody reads the last five before writing the sixth. The single most valuable habit in this whole area, and almost nobody does it. Repeats are the signal — the same contributing condition showing up three times means the fix from last time did not happen or did not work, and that pattern is invisible one document at a time. It is the same reason a decision journal only works if you reread it.

Which means the reviews have to be findable by content, not filed by date. You will search for "cache" or "rollback", not for "the March one".

The personal version

Not everyone works somewhere with a process. The same shape works for one person.

Anything that went badly and took real time — a bad deploy, a lost afternoon, a data problem you had to unpick — is worth six lines: what happened, what you thought at the time, what actually caused it, what made it hard to fix, one thing to change. Ten minutes, once.

A year of those is a remarkably good picture of how your own systems fail, and it is the material that turns "I have been doing this for five years" into something more useful than time served.

The honest version

If you run incidents at any scale, use an incident management tool. Timeline capture, paging, status pages and action tracking are a real product category and they do things a notes app will not.

For a small team, a solo operator, or the personal version, the shape above is most of the value and it needs a text editor and the discipline to write during rather than after.

Cyanote has incident review and retro templates in the / menu — the five sections in order, so the document starts as prompts rather than a blank page. Notes hold timestamped lines, code blocks with the actual error, and images of the graph, all in one document; ⇧⌘F searches the body of every review you have written, which is what makes "have we seen this before" a question you can answer in four seconds. It is one local database on your own Mac, which for the personal version is the point — your record of how things fail is yours, not your employer's.