Back

Blameless postmortems are a method for getting accurate facts, and the test is what changes

·21 min read

In the summary Amazon Web Services published after the S3 disruption of 28 February 2017, the trigger fits in one sentence. At 9:37 AM Pacific time an authorized team member, following an established playbook, ran a command to remove a small number of servers, and one input was entered incorrectly, so far more servers went than intended. Two subsystems that S3 needs to serve requests had to restart. By 12:26 PM the index subsystem had enough capacity back to serve GET, LIST and DELETE requests, it was fully recovered at 1:18 PM, and the placement subsystem, which PUT requests also need, finished recovering at 1:54 PM. AWS's list of corrective actions was about the tooling and the architecture: the tool now removes capacity more slowly, it refuses removals that would take a subsystem below its minimum required capacity, and other operational tools were to be audited for similar checks. The document names no one and does not ask the engineer to be more careful.

My claim is that blamelessness is an instrument, and the rest of this essay defends that reading. It is a method for collecting accurate information from people who have good reasons to withhold it. Organizations choose it because punishment changes what people say, and a postmortem is only as good as what people say. The method has a product, which is a timeline, a set of contributing factors and follow-up actions with owners, and it has a test, which is whether something in the system is different a month later. By that test some reviews that call themselves blameless fail, and I expect the failure to sit in the action list. I have no count of how often.

The same logic applies to AI incidents, where two blame targets are on offer: "the model" and "the person who wrote the prompt". Both explain little, and I will show why with a tribunal decision about a chatbot and three vendor postmortems about model quality. The evidence for the method comes first, including the places where it is thin.

People report what they are not punished for reporting

The empirical case starts in hospitals. Amy Edmondson set out to test an expectation that sounds obvious: hospital units with better teamwork should make fewer medication errors. In a 1996 paper in the Journal of Applied Behavioral Science she reported the opposite pattern in the recorded data, from eight nursing units. According to the AHRQ's PSNet summary of the paper, units with stronger nurse-manager direction and coaching and better unit relationships had higher detected error rates. Her explanation is that a detected rate depends on two quantities: how many errors occur and how many get reported. Harvard Magazine's 2001 account says the units ranged from authoritarian and punitive to open and supportive, and that detected error rates had an almost direct linear relationship to openness. A summary of the paper on the Field Guide site adds that a second researcher, blind to the error data, observed each unit and ranked them on openness, and that this ranking matched the error-rate ranking almost unit for unit. I read the paper only through these secondary accounts.

The inference has limits. With eight units the analysis rests on eight data points, units differ in many ways, and unreported errors cannot be counted by definition. What the study established is narrower than the slogan it became. A recorded error count says little about safety until you know the reporting climate, and a manager who disciplined the unit with the most reports would be disciplining the most open unit.

Three other sources point the same way. Edmondson's 1999 paper in Administrative Science Quarterly studied 51 teams in a manufacturing company and defined psychological safety as a shared belief that the team is safe for interpersonal risk taking. It found that this belief was linked to learning behavior, and that learning behavior was linked to team performance. Lucian Leape, testifying to Congress in 2000, in words quoted in the American Nurses Association's 2010 position statement on just culture, said that punishing individuals gives people strong incentives to report only the errors they cannot hide. James Reason, in a 2000 BMJ article, tied effective risk management to a reporting culture, which depends on trust, which in turn requires a just culture with a collective understanding of where the line falls between blameless and blameworthy actions. For scale he cites a 1997 source for the figure that some 90 percent of quality lapses in aviation maintenance were judged blameless.

John Allspaw put the same logic in engineering terms at Etsy in 2012. In the post as archived by the Internet Archive, he argues that an engineer who expects a reprimand is discouraged from giving the details needed to understand a failure, that trust between engineers and management erodes, and that latent conditions for failure stay unidentified because people go silent. The same failures then come back. His corrective is to ask engineers for a full account of their actions, observations, expectations and assumptions, and to treat them as the best experts on their own mistakes.

The product of a postmortem is a timeline, contributing factors and actions with owners

Google's SRE book describes a postmortem as a written record of an incident: its impact, the steps taken to mitigate or resolve it, the root causes, and follow-up actions to prevent recurrence. It also says teams should define in advance what triggers one, with examples such as user-visible downtime above a threshold, any data loss, an on-call intervention like a rollback, and a monitoring failure. Triggers fixed in advance keep the decision to write a postmortem apart from any verdict about a person. Reviewers are told to check whether the action items are appropriate and prioritized.

The three outputs do different work. A timeline is the evidence, and it has to be built forward, from what the people involved could see at each step, because hindsight makes the right move look obvious. Sidney Dekker's field guide, in the August 2000 draft I could read, frames the investigator's task as finding how people's assessments and actions made sense at the time, given the circumstances around them, and Allspaw's Etsy post builds on the same idea. Contributing factors are the explanation, and the plural is deliberate, since a single root cause invites a single culprit. Only the actions can change what happens next, which is why I count them as the product of the exercise.

The AWS summary shows what the actions look like when the method works. AWS changed the tool to remove capacity more slowly and added a check against removals below a subsystem's minimum capacity. It started auditing other tools for the same gap, and it reprioritized work to partition the index subsystem into more cells, to improve recovery time. It also changed the status dashboard's administration console to run across multiple regions, since its dependency on S3 had blocked status updates until 11:37 AM. Each can be marked done or not done, and none depends on a person remembering something at the right moment. The mistyped input was still the proximate trigger. The review treated it as an event the system should have been built to survive.

Etsy's practice, as InfoQ described it in 2014, made postmortem action items take priority over other work, including shipping product, which is one way to make the test of change binding.

Fred Hebert, writing for the Resilience in Software Foundation in April 2026, names the failure mode superficial blamelessness. Retribution is avoided, yet the remediation still centers on individuals: more training, reminders to pay attention, one more reviewer. His piece is an argument from practice with no data, so I use it as a checklist and not as a finding. One heuristic in it is still usable: if most of the actions are about what specific people should do better, suspect that the review stopped at the individuals, and ask of each item whether it changes the conditions under which the next person will decide.

There is a gap in the evidence. The sources I opened establish a mechanism, that punishment suppresses reporting, and they include practitioner accounts of how postmortems are run. I did not find a controlled study showing that organizations which adopt blameless postmortems have fewer or shorter incidents. What exists is a plausible mechanism plus consistent practice.

The accountability objection is half right

The strongest objection says that blamelessness removes the pressure that makes people careful. If a team knows that a careless deployment will produce a polite document and no consequence, carelessness becomes cheaper. A second version is more cynical and, I think, closer to how the method fails in practice. Organizations adopt the word to protect senior people, so that decisions about staffing, deadlines and risk acceptance are never examined while an engineer's keystrokes are examined in detail.

The method has a reply to the first version, and the reply is in the tradition it comes from. David Marx's just culture model, as summarized in the ANA statement, separates three kinds of behavior. Simple human error, doing something other than what was intended, calls for consoling the person and changing processes, procedures, training or design. At-risk behavior is a choice that raises risk where the risk is not recognized or is wrongly believed to be justified, and it calls for coaching and for removing the incentives that reward the shortcut. Reckless behavior is conscious disregard of a substantial and unjustifiable risk, and it calls for remedial or punitive action. In the ANA's summary, Marx argued that discipline should be tied to the behavior and the risk it carried more than to the actual outcome. The statement adds that this differs from a culture whose governing principle is no blame, because a just culture does not tolerate conscious disregard of clear risks.

The clause about outcomes is the part I find most persuasive. Take the S3 case: a mistyped input is the same behavior whether it removes ten servers nobody misses or the servers a region depends on, and a rule that punishes only the second punishes luck. Accountability in the just-culture sense attaches to what the person knew and chose, and it can be applied consistently only when the review has first established what that was, which is the reason the review needs candid accounts.

Allspaw's version adds a forward-looking meaning of the word. In his account, engineers are accountable by giving a detailed account of their contribution and by helping to design and teach the fix. The person who made the error ends up owning part of the follow-up, which is a heavier load than a reprimand.

The second version of the objection is right, though. Blamelessness applied only at the sharp end moves accountability to a place nobody looks. At-risk behavior is the category that tests this, because it is usually produced by incentives, such as the shortcut that saves an hour and is praised for it. Marx's remedy is to remove those incentives, which is a manager's task. A postmortem that coaches the engineer and leaves the incentive in place has completed half of the prescribed response, and Hebert's list of individual-focused remediations is a list of ways that happens. A defensible version of the method is therefore narrower than "nobody is to blame". It says that blame should not be the mechanism for collecting facts, that sanctions belong to a separate process with its own standard, and that the organization's own decisions are inside the scope of the review.

That reply has a limit. Reckless behavior exists as a category on paper, and I do not know how often organizations use it. A line that is never invoked is hard to distinguish from a line that does not exist, and I found no data on how often anyone crosses it in practice.

A chatbot's wrong promise belongs to the company that deployed it

In November 2022 Jake Moffatt asked Air Canada's website chatbot about bereavement fares while arranging travel. According to press and law-firm coverage, since I could not open the decision itself, the chatbot said he could book at the regular price and claim the bereavement fare afterward, by applying within 90 days of the ticket's issue date. The reply linked to a page that said the opposite, that the bereavement policy did not apply to requests made after travel. His application was unsuccessful, Air Canada staff told him retroactive applications were not allowed, and in February 2023 an employee told him the chatbot had used misleading words. On 14 February 2024 the Civil Resolution Tribunal of British Columbia, in Moffatt v. Air Canada, 2024 BCCRT 149, ordered the airline to pay $650.88 in damages for negligent misrepresentation, $36.14 in interest and $125 in tribunal fees.

Air Canada's defense is the part that matters here. It argued, in effect, that the chatbot was a separate legal entity responsible for its own actions, and that the correct policy sat on a linked page that the customer should have trusted over the chatbot. The tribunal rejected both. It treated the chatbot as part of the airline's website, saw no difference between a static page and a chatbot, found no reason a customer should have to cross-check one against the other, and held that Air Canada had not taken reasonable care to make sure the chatbot was accurate. The amount at stake was a few hundred dollars, and I did not check how later tribunals treat the ruling, so it is one adjudicator's reasoning and not settled law. What it shows is what happened, in one legal setting, when an organization tried to disown its own system's output.

A blameless review of that incident would have to answer questions the decision leaves open. It does not describe how Air Canada built the bot, so I can only list the questions, and they are the ones the separate-entity defense skips. A timeline would start with the policy text and the date the bot first gave the 90-day answer, and not with the customer's chat. Among the contributing factors to examine would be where the bot's answer came from, whether anything compared it with the policy page it linked to, who owned the consistency between policy and bot, and whether bereavement fares, a policy question with money attached, appeared in any test set. Last would come what happened after the employee's February 2023 admission, because that was a documented moment when the company knew its bot had said something its own policy contradicted. Each of these has an owner who can be given a date, and none requires deciding that a particular person was negligent.

Now take the two blame targets in turn. "The model hallucinated" describes the symptom. For a probabilistic text generator, producing a plausible wrong answer at some rate is expected behavior, so treating it as the cause is the AI equivalent of writing "operator error" in the report: it names the surface event and ends the inquiry. "Whoever wrote the prompt" moves the blame to a person whose instructions could not by themselves guarantee any particular answer, and it teaches the next person to stay away from prompts. Both leave the real contributing factors, the missing check between answer and policy and the missing owner, untouched. That is my judgment about where the fix belongs, and the review should still record how prompts and models failed.

Silent model changes call for a review that starts before the incident

A second kind of AI incident arrives from outside. The model behind a product changes, the product gets worse, and nothing visible changed on the customer's side. Vendors' own postmortems are worth reading as examples of the genre. They are also the vendor's account, published selectively and written for the vendor's purposes.

On 25 April 2025 OpenAI updated GPT-4o in ChatGPT, and the model became noticeably more sycophantic. OpenAI pushed a system prompt change that weekend and rolled the update back on the Monday. In its postmortem of 2 May, as Simon Willison summarized it, offline evaluations looked good and small A/B tests suggested users liked the model. Internal expert testers said the behavior felt slightly off, and the quantitative signals outweighed that impression. The update had not been announced because OpenAI expected it to be subtle. OpenAI committed to treating behavior problems such as personality as launch-blocking, and to communicating updates even when they seem small. I read this postmortem through Willison's account and not in full.

Anthropic's postmortem of 17 September 2025 covers three infrastructure bugs from August and early September. One sent some Sonnet 4 requests to servers configured for a longer context window. It affected 0.8 percent of requests at first and 16 percent in the worst hour, on 31 August, and about 30 percent of Claude Code users in the period had at least one message misrouted. Detection was slow, the post says, because ordinary evaluations did not capture the degradation, because privacy controls limited engineers' access to user interactions unless users submitted them as feedback, and because different bugs produced different symptoms on different platforms. The company also wrote that it had relied too heavily on noisy evaluations and had not tied the early reports to a recent load-balancing change.

Anthropic's April 2026 update on Claude Code traced complaints to three changes. A lower default reasoning effort ran from 4 March to 7 April. Between 26 March and 10 April a caching bug, in a change meant to clear old reasoning only from sessions idle for over an hour, cleared it on every turn. A system prompt instruction limiting verbosity ran from 16 to 20 April. For the last, weeks of internal testing showed no regression on the evaluations run at the time, and a broader set used afterward showed a 3 percent drop for two models. As I read them, none of the three documents attributes its incident to a named person. They name changes, thresholds, tests and signals, and the commitments are mostly to release gates such as per-model evaluations, ablations of each prompt line, soak periods and gradual rollouts.

For a company that consumes a model through an API or a product, the vendor's postmortem is an input to your review and cannot replace it. Your own review should open with the question the vendors' reviews keep returning to: why did the instrument say everything was fine? A made-up example shows how that can happen without anyone being careless. Suppose a nightly evaluation of 200 cases passes 90 percent, and a vendor change drops the true pass rate to 85 percent. Comparing one night's score against the previous one, and treating the two nights as independent samples, the standard error of the difference is about 3.3 points, so a five-point drop sits at roughly 1.5 standard errors and cannot be told from noise. Reaching 2.6 standard errors, about the usual cutoff for a 1 percent two-sided test, for the same drop needs about 600 cases per night. The 600 is the point where the expected drop just reaches that cutoff, so reliable detection would need more. If the same 200 cases are rerun every night, a paired comparison is less noisy, but then the set speaks only for those 200 cases. An on-call engineer who looked at the dashboard and saw nothing did what the instrument allowed. The fault sits in a test set too small to show a five-point drop, and that is where the action item goes.

A review of such an incident, whether the cause was a vendor change, our own prompt or both, would contain these items:

  1. A timeline built from our own logs, with the model identifier, prompt version, retrieval index version and any vendor notices placed beside the user-visible events.
  2. Detection: what noticed the problem, how long after the first bad output, and which monitors stayed silent and why.
  3. Contributing factors by layer: vendor model or serving change, our prompt and configuration, data and retrieval, guardrails, and the human process around them.
  4. The decision points, with what each person knew at the time and why the action made sense to them.
  5. Customer impact, and anything the system promised that the company must now honor or correct.
  6. A classification of the behaviors involved as error, at-risk behavior or recklessness, with the incentives that shaped any at-risk choices.
  7. Action items with an owner, a date and a stated way to verify completion, at least one of which changes detection.
  8. What the vendor was asked for, such as change notices, version pinning or a regression report, and what we promised to tell our own customers.

The test of a postmortem is a change you can point to

Take the last three postmortems your team wrote. For every action item, record the owner, the date it was finished and how anyone would know it worked. Count how many items alter a tool, a check, a default or a permission, and how many say to train, remind or be careful. Then search the incident tracker for repeats of the same failure class, since a repeat is the plainest evidence that an earlier action changed nothing.

If an AI system is in production, add one more exercise. Pick the vendor model it depends on, assume it silently regressed by five points on your own task last night, and find out how long your current monitoring would take to notice. Whatever the answer is, it is the first action item of the next review, and it can have an owner before the incident happens.

Sources

  1. Amazon Web Services (28 February 2017). Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. https://aws.amazon.com/message/41926/
  2. Edmondson, A. C. (1996). Learning from mistakes is easier said than done: Group and organizational influences on the detection and correction of human error. Journal of Applied Behavioral Science 32(1), 5-28. The full text was not opened. Read through the AHRQ PSNet summary (https://psnet.ahrq.gov/issue/learning-mistakes-easier-said-done-group-and-organizational-influences-detection-and), Harvard Magazine (2001), Secret Errors Kill (https://www.harvardmagazine.com/node/542), and a summary on the Field Guide site (https://explore.psychsafety.com/n/edmondson-1996.md), which is secondary commentary.
  3. Edmondson, A. C. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly 44(2), 350-383. Abstract read via the Harvard DASH record (https://dash.harvard.edu/entities/publication/13a7b031-0fdd-45ec-a7e0-2b80e2bc679f); the full text was not opened.
  4. Reason, J. (2000). Human error: models and management. BMJ 320, 768-770. https://pmc.ncbi.nlm.nih.gov/articles/PMC1117770/
  5. Allspaw, J. (22 May 2012). Blameless PostMortems and a Just Culture. Etsy, Code as Craft. https://www.etsy.com/codeascraft/blameless-postmortems/ (the Etsy page blocked automated access, so the text was read from the Internet Archive copy of that page).
  6. Google (2016). Postmortem Culture: Learning from Failure. Chapter 15 in Beyer, B., Jones, C., Petoff, J., Murphy, N. R. (eds.), Site Reliability Engineering. https://sre.google/sre-book/postmortem-culture/
  7. American Nurses Association (28 January 2010). Position Statement: Just Culture. https://anaprodsite1.nursingworld.org/globalassets/practiceandpolicy/health-and-safety/just_culture.pdf. The statement was read in full. It summarizes Marx, D. (2001), Patient Safety and the "Just Culture": A Primer for Health Care Executives (Columbia University), and quotes Leape's testimony of 25 January 2000 to a US Senate subcommittee; the Marx primer and the testimony were read only through this statement.
  8. Dekker, S. (2006; 3rd ed. 2014). The Field Guide to Understanding 'Human Error'. Ashgate. Library catalogues describe the 2006 book as a revised edition of his 2002 Field Guide to Human Error Investigations. I did not read the published editions. The sentence cited in the text comes from Dekker's draft of August 2000, The Field Guide to Human Error (https://www.leonardo-in-flight.nl/PDF/FieldGuide%20to%20Human%20Error.PDF), whose wording may differ from the published books.
  9. Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal of British Columbia, 14 February 2024). The decision text on the tribunal's site and on CanLII blocked automated access and was not read. Facts, quotations and the award (damages $650.88, interest $36.14, tribunal fees $125) were read through Daily Hive (15 February 2024, https://dailyhive.com/canada/air-canada-chatbot-decision) and a law-firm article on dww.com (https://www.dww.com/articles/bc-tribunal-finds-air-canada-liable-for-inaccurate-advice-given-by-website-chatbot), plus search-result summaries of other law-firm coverage.
  10. Anthropic (17 September 2025). A postmortem of three recent issues. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
  11. Anthropic (23 April 2026). An update on recent Claude Code quality reports. https://www.anthropic.com/engineering/april-23-postmortem
  12. OpenAI (2 May 2025). Expanding on sycophancy. https://openai.com/index/expanding-on-sycophancy/ (read through Willison, S., 2 May 2025, https://simonwillison.net/2025/May/2/what-we-missed-with-sycophancy/).
  13. Hebert, F. (15 April 2026). Superficial blamelessness. Resilience in Software Foundation. https://resilienceinsoftware.org/news/11502437
  14. Miranda, J. (28 July 2014). Blameless post-mortems. InfoQ. https://www.infoq.com/news/2014/07/blameless-post-mortems
  15. Illustrative evaluation arithmetic: a nightly set of 200 cases, a true pass rate falling from 90 to 85 percent, a standard error of about 3.3 points for the difference, and about 600 cases for 2.6 standard errors. These numbers are made up for the example.