Postmortems and corrective actions: evidence, contributing conditions and verification
Build an evidence-based incident review and turn contributing conditions into owned, verifiable actions. Distinguish prevention, detection and mitigation improvements without blaming the last operator.
TL;DR: A postmortem should explain how the system allowed the incident and which changes would reduce recurrence or impact. Ground the timeline in evidence, preserve uncertainty and give each selected action an owner and a testable completion condition.
Reconstruct what responders could know
Separate observed events from later interpretation. A deployment timestamp, an alert, an operator command and the first customer report provide different evidence. Record time zones and clock limitations before concluding that one event preceded another by a few seconds.
Explain the information available when each decision was made. A choice that looks poor after diagnosis may have been reasonable under the evidence available during the incident. Google's postmortem culture chapter describes blameless analysis and learning from contributing conditions.
Blamelessness does not remove responsibility for follow-up work. It changes the question from who last touched the system to how design, access, workload and operational context shaped the outcome.
Connect conditions to controls
Prevention is valuable, but a system may also need better detection and safer mitigation. A single fix to the triggering bug can leave the response weaknesses untouched.
Worked analysis: a connection-limit change
Imagine a deployment increased application replicas. Each replica retained the previous database pool size, so total possible connections exceeded the database's practical limit. A CPU alert did not fire, and responders initially added more replicas to address timeouts, worsening the condition.
The trigger was the replica increase. Contributing conditions included per-instance pool configuration without a fleet-level budget, insufficient connection telemetry and an autoscaling response that assumed CPU was the bottleneck. Calling the cause “an engineer chose the wrong replica count” would fail to explain why the configuration and response process permitted the outcome.
Useful actions might include enforcing a total connection budget in deployment review, exposing saturation and waiting-time evidence, and testing the scale-out behavior under a constrained database. These actions target different mechanisms. Choose the ones with enough expected benefit to justify implementation and maintenance.
Do not manufacture an exact recovery-time improvement before the changed process has been exercised. Record a target and later attach measured evidence from an exercise or subsequent relevant event.
| Weak action | Stronger, verifiable action | Completion evidence |
|---|---|---|
| Be more careful scaling | Add a checked connection-budget calculation to the deployment path | An unsafe test configuration is rejected |
| Improve monitoring | Alert on a defined user-impact condition with database context | Controlled failure reaches the correct responder |
| Update the runbook | Add a decision for connection saturation and a safe mitigation | A responder completes a rehearsal using it |
| Train the team | Practice the identified unfamiliar recovery operation | Exercise notes show correct execution and gaps |
A timeline is not yet a causal explanation
Temporal association can identify candidates, but a change occurring before an incident does not prove it caused the incident. Compare affected and unaffected cohorts, inspect the mechanism and record what happened after mitigation. Production change correlation develops that reasoning.
Use qualified statements when evidence is incomplete. “The pool increase is the leading explanation because connection demand rose with replicas and recovered after reducing concurrency” is stronger than an invented certainty about every internal event.
Avoid forcing one root cause when several controls independently failed. A dependency outage may trigger user impact only because retry amplification, missing capacity limits and a broken alert route combine. The review should preserve that causal structure.
Manage actions as product work
Give each action an accountable owner, priority and completion condition. Link it to the condition it addresses. A large unprioritized list makes the review look thorough while allowing the most useful work to remain unfinished.
Review overdue items and explicitly accept or reject remaining risk. Closing an action because a document exists is insufficient when the action promised a working recovery capability. Test the capability at the relevant boundary.
Protect sensitive incident details. Use sanitized examples in public writing and interviews, and distinguish your personal contribution from the team's result. Project narratives explains how to discuss real evidence without inventing metrics or exposing confidential information.
Is “human error” a sufficient cause? No. Describe the action and the system conditions that made its harmful effect possible, including missing feedback or constraints.
Must every incident produce a code change? No. The best action may repair ownership, a dependency agreement or an operational procedure. It still needs a verifiable outcome.
When should a postmortem say the cause is unknown? When the evidence cannot support a stronger statement. Preserve leading hypotheses and identify useful evidence collection rather than filling the gap with a plausible story.