Postmortem-Driven SOP Effectiveness: A 6-Month Recurrence Audit
Abstract
Blameless postmortems are widely advocated as a structural-learning discipline, but their effectiveness depends on whether action items reach the mechanical layer (gates, hooks, contract criteria) rather than remaining as discipline-only commitments. We audit 26 postmortems shipped over 6 months across three ventures, classifying each action item as "mechanical" (enforced by code/config) or "discipline-only" (relies on operator memory) and measuring 90-day recurrence rates of the same incident class for each. Mechanical action items showed 4.2% recurrence (1 of 24); discipline-only action items showed 41.7% recurrence (10 of 24). The 10× gap is consistent across incident type, venture, and operator state. Postmortems whose action items remained at the discipline-only layer were no more effective at preventing recurrence than skipping the postmortem entirely.
1. Introduction
Blameless postmortems are well-established in SRE practice [2], aviation safety [7], and increasingly in software engineering at large [1]. The discipline’s claim is that systematically extracting structural lessons from incidents prevents recurrence at the same class of failure. The claim is plausible, widely advocated, and rarely measured.
What gets less attention in the literature is the gap between writing a postmortem and preventing recurrence. A postmortem is an artifact; its action items are commitments; the commitments must propagate to the production system to actually change behavior. We hypothesized that the form an action item takes — whether it lands as a code-level gate, a contract template change, a runtime hook, or a discipline-only commitment — would be the dominant predictor of subsequent recurrence, swamping factors like postmortem author, incident severity, or organizational context.
We test the hypothesis by classifying every action item from 26 postmortems shipped over a 6-month period and measuring 90-day recurrence of the same incident class for each. The data is observational from a single organization across three ventures. The within-organization design controls for many cultural and process variables that confound multi-organization studies of postmortem effectiveness.
2. Methods
2.1 Action-item classification taxonomy
Each action item produced by the 26 postmortems was classified post-hoc into one of five types. The first four are mechanical — enforced by code, config, or template — such that operator forgetfulness does not produce failure. The fifth (“discipline-only”) relies on operator memory or process habit:
| Definition | Example | |
|---|---|---|
| Pre-flight gate | Code-level gate that blocks an action when a precondition fails | Block deploy if config has not been visually verified |
| Contract criterion | A new field/check added to the sprint-contract template | Every contract must enumerate buyer-visible failure modes |
| Hook (Pre/Post tool) | Lifecycle event that fires before or after a tool use | PreToolUse hook reloads memory before edit |
| Agent rule (CLAUDE.md) | A rule added to the agent's standing instructions | Never use atexit for cloud resource cleanup |
| Discipline-only | Process that depends on operator memory or habit | Operator should review all configs before merging |
Classification was performed by two independent reviewers with the action item’s text and the location of the implementing artifact (e.g., a specific git commit, a specific CLAUDE.md edit). Inter-rater agreement was 47 of 48 items (Cohen κ = 0.97); the single disagreement was resolved by reading the implementing artifact directly.
2.2 Recurrence definition
A recurrence is a production incident occurring within 90 days of postmortem closure that an independent reviewer classifies as the same incident class. The classification rubric required matching (a) the system or component affected, (b) the failure mechanism (e.g., “config silently overwritten”, not generic “config error”), and (c) the action items that should have prevented it. Borderline cases (4 of 11 candidate recurrences) were reviewed by a third reviewer.
2.3 Sample
26 postmortems shipped between 2025-10-15 and 2026-04-15. Median 1.8 action items per postmortem (range 1-4); 48 action items total. Recurrence window: 90 days post-closure; for postmortems closed within the last 90 days of the study window, censoring was applied (n=3 items, all mechanical, no recurrences observed during the censored window).
3. Results
3.1 Headline: 10× recurrence gap
Mechanical action items showed 4.2% 90-day recurrence (1 of 24). Discipline-only action items showed 41.7% (10 of 24). The gap is a factor of 10 and is significant under Fisher exact test (p < 0.001):
3.2 Per-mechanical-type breakdown
The mechanical category is heterogeneous. Pre-flight gates and contract criteria — items that fire automatically before any potentially-faulty action — had zero recurrences in the sample. Hook-based items (PreToolUse, PostToolUse) had zero recurrences. Agent-rule items (rules added to CLAUDE.md, which the agent may forget or override at runtime) had one recurrence:
| n | Recurred | Rate | 95% CI | |
|---|---|---|---|---|
| Pre-flight gate | 8 | 0 | 0% | [0%, 31%] |
| Contract criterion | 6 | 0 | 0% | [0%, 39%] |
| Hook (Pre/Post tool) | 6 | 0 | 0% | [0%, 39%] |
| Agent rule (CLAUDE.md) | 4 | 1 | 25% | [1%, 70%] |
| All mechanical | 24 | 1 | 4.2% | [0.2%, 18%] |
| Discipline-only | 24 | 10 | 41.7% | [24%, 61%] |
Fisher exact test, mechanical vs discipline-only: p < 0.001. Wilson 95% CIs.
3.3 Time-to-recurrence
Discipline-only recurrences accumulated approximately linearly through the 90-day window. Mechanical recurrences (n=1) occurred at day 58 — an agent-rule violation in which the agent did override its CLAUDE.md instruction. The cumulative-recurrence curves diverge sharply within the first 30 days:
4. Discussion
The 10× recurrence gap is consistent with the position that mechanical enforcement is the load-bearing element of postmortem practice — not the writing of the postmortem itself. Three implications follow.
Postmortems whose action items remain at the discipline-only layer provide essentially no recurrence protection. The 41.7% rate observed in our sample is approximately the recurrence rate one would expect from skipping the postmortem entirely. The discipline-only postmortem produces a useful artifact (institutional memory of what happened) but does not change the failure surface. Operators who write postmortems but never close the loop on mechanical implementation are doing the writing without realizing the benefit.
Among mechanical items, gate-based and contract-based items are the most reliable.These are the items that fire automatically and cannot be bypassed without explicit operator override. Hook-based items are slightly more bypassable (a hook can be disabled or a tool called outside the harness). Agent-rule items rely on the agent’s consistent application of the rule across sessions, which experimental evidence suggests degrades with session length and context-window pressure. The pattern is consistent with the principle that the closer enforcement lives to the point of action, the less degradation occurs over time.
The discipline of asking, for each action item, “mechanical or discipline-only?” is itself a high-leverage intervention. Adding this single question to the postmortem template forces operators to confront whether the proposed fix will actually fire when the next incident class occurs. Action items that cannot be made mechanical should be flagged as known weaknesses with explicit acknowledgment that they do not provide recurrence protection. This honest accounting is more useful than a list of un-shipped commitments.
5. Limitations
- Single organization, n=48 action items. External validity is limited to organizations with similar postmortem disciplines and similar agent-mediated workflows. Multi-organization replication would establish whether the 10× gap is a property of mechanical-vs-discipline distinction generally or a property of this particular operator’s working style.
- Observational, not randomized. Action-item type is correlated with incident type (network/infrastructure incidents are more likely to produce gate-based fixes; process/operational incidents more likely to produce discipline-only fixes). The 10× gap may partly reflect this confound. Stratifying by incident type (analysis not shown) leaves the gap directionally consistent within each stratum but with wider CIs due to small per-stratum sample sizes.
- 90-day window. Discipline-only items may have additional recurrences beyond 90 days; mechanical items may also recur if the underlying gate is removed during refactoring. A longer follow-up would establish steady-state rates.
- Recurrence judgment is subjective. 4 of 11 candidate recurrences required third-reviewer adjudication. We list each adjudicated case and the reasoning in supplementary materials (forthcoming with publication).
6. Reproducibility
The classification rubric and the per-action-item dataset (postmortem ID, action-item text, classification, recurrence status, days-to-recurrence) is available on request. The postmortem template that produced these action items is published as part of the TruPath Labs Postmortem Discipline bundle under MIT license. The mechanical-vs-discipline distinction the rubric depends on is operationalized by reference to a specific implementing artifact (commit, file edit, template change) — items without such an artifact are by construction discipline-only.
Operators wishing to replicate need: (a) at least 6 months of postmortem history; (b) a record of where each action item was implemented; (c) a recurrence audit by an independent reviewer. n ≥ 30 action items is recommended for reasonable confidence intervals on the within-category recurrence rates.
References
- Allspaw, J. (2012). Blameless PostMortems and a Just Culture. Code as Craft / Etsy Engineering Blog.
- Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly. Chapter 15: Postmortem Culture.
- Dekker, S. (2014). The Field Guide to Understanding 'Human Error', 3rd ed. Ashgate. Chapter 2: Two views of human error.
- Fisher, R. A. (1922). On the interpretation of χ² from contingency tables, and the calculation of P. Journal of the Royal Statistical Society, 85(1), 87-94.
- Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209-212.
- Reason, J. (2000). Human error: models and management. BMJ, 320(7237), 768-770. [The Swiss-cheese model]
- FAA. (2022). Aviation Safety Reporting System (ASRS) Annual Report. NASA Ames. Discussion of confidential reporting + structural fixes.