Operational runbooks: decision steps, stop conditions and rehearsal evidence
Create operational runbooks with explicit triggers, expected observations, authority and stop conditions. Rehearse them with an unfamiliar responder and keep evidence of what the procedure actually verifies.
TL;DR: A useful runbook lets another responder decide whether the procedure applies, execute a bounded action and verify the outcome. Validate those decisions in a rehearsal; a reviewed document full of commands can still fail at the first unexpected result.
Write for the responder who did not build the service
An architecture document explains how a system works. A runbook guides a repeatable operational procedure. A diagnostic playbook helps choose between hypotheses when the cause is still uncertain. Teams use these names differently, so define the document's purpose at its start instead of relying on the label.
For a procedure, name the service and environment, trigger, required access and authority to make changes. Include a quick applicability check. A database recovery procedure for one replication topology should stop when the operator discovers a different topology, rather than encourage them to adapt destructive commands under incident pressure.
AWS's runbook guidance connects procedures to prerequisites, steps and expected outcomes. Google's on-call workbook describes operational preparation and exercises. Use those ideas to test the document's decisions, not to decorate it with a long generic checklist.
Replace a restart instruction with a bounded decision
Consider a fictional worker service whose oldest queued job is now twenty minutes old. The old page says “restart workers if the queue grows.” That procedure cannot distinguish dead workers from a downstream outage, and restarting all workers could erase the only useful local evidence.
The replacement procedure first checks that the queue and service match the alert. It then compares worker activity, completion rate and dependency errors. If workers are consuming and failing because the database rejects connections, the restart branch does not apply. Escalate to the dependency owner and contain retries using the approved service procedure.
The uncertainty branch matters. A runbook should tell the operator when its assumptions no longer hold.
| Procedure element | Concrete content for this case | What a rehearsal should expose |
|---|---|---|
| Applicability | Queue identity, environment and deployment | Similar names lead to the wrong service |
| Observation | Completion rate and dependency failures | Dashboard permission is missing |
| Action boundary | One worker, approval and retained evidence | Operator assumes all workers should restart |
| Verification | Queue age, successful jobs and duplicate effects | Process health improves but work still fails |
Make expected results interpretable
A command should have an expected observation and a branch for an unexpected result. Avoid embedding a production identifier in a copyable destructive command. Require the operator to obtain the intended resource identity from a trustworthy inventory and verify it before execution. Keep access instructions current enough that the responder can reach the required evidence.
For the fictional case, suppose one hundred jobs are waiting, new jobs arrive at ten per minute, and recovered workers complete thirty per minute. Net drainage is twenty per minute, suggesting about five minutes to clear that fixed backlog under steady conditions. A falling worker error rate with no successful completions does not satisfy this recovery check. The estimate also changes if arrivals rise or job duration varies.
The runbook should state which signals must improve and how long to observe before escalating. Those limits come from the service's risk and workload, not a universal five-minute rule. Google's incident-response guidance also separates operational response from communication and coordination; the document should identify the handoff owner rather than leave that responsibility implicit.
Rehearse with someone who has not memorized it
Use a disposable environment or a tabletop with sanitized evidence. Ask a responder unfamiliar with the procedure to explain each decision and expected result before acting. Include one case where the documented action is appropriate and another where it must stop. Record where they needed undocumented knowledge, unavailable access or a guess about the output.
Repair those specific gaps and repeat the affected branch. Track the document owner, service/configuration versions and last exercised scenario. A fresh edit timestamp proves that a file changed; it does not prove that recovery still works. Link the exercise record and unresolved gaps so a future responder can judge the procedure's limits.
In an interview, use a real documentation change you made and explain what another engineer could do afterward. If you observed fewer clarification requests but did not measure incident duration, report that observation. Avoid turning one rehearsal into a claimed organization-wide reliability gain.
Self-check: a runbook's commands all succeed, but the business operation remains unavailable. Has the procedure passed? No. Its verification contract is incomplete or the assumed failure was wrong. Stop repeating successful commands, preserve the observations and follow the escalation branch while updating the missing recovery check.