Mitigate before you diagnose
Understanding a failure and stopping it are different goals, and during an incident only one of them is urgent. The instinct to find the cause first is what turns a five-minute outage into an hour, and it is a useful failure mode to practice avoiding in an incident round.
TL;DR: Restore service with the fastest safe action available, then diagnose with the pressure off. Use a rehearsed rollback, failover, load-shedding rule or feature flag when its safety conditions hold. These actions can reduce impact before you know the root cause. Diagnosis is what the postmortem is for.
The two goals compete
An engineer looking at a broken system wants to understand it. That instinct is correct in every context except this one. While you read logs, users are still failing, and the error budget is still burning.
The discipline is to separate the questions. What stops this now is answerable in seconds from a short list. Why did this happen takes as long as it takes, and it can happen after the graphs recover.
The mitigation menu
These common mitigations may work before full root-cause analysis, but each requires enough diagnosis to establish its safety conditions:
- Roll back. If the timing correlates with a deploy and rollback is compatible with current state, reverse it. You do not need to know which line broke.
- Fail over. Move traffic to the healthy region, zone, or replica set.
- Shed load. Drop the lowest-value traffic so the highest-value traffic survives. Better than everything failing equally.
- Disable the feature. A flag is a mitigation, which is the strongest argument for having them.
- Scale up, when the constraint is capacity and capacity is available.
- Break the loop. Cascading failures need the retry storm or the feedback cycle interrupted, which is often a config change rather than a fix.
Keep these options in the runbook, with the conditions that make each safe.
When diagnosis has to come first
The rule is not absolute, and knowing the exceptions is what makes it a judgement rather than a slogan:
- Data corruption. Rolling back code while a bad writer keeps running can make the damage worse. Establish what is being written before you act.
- A rollback that is destructive. If the deploy removed a column or changed a stored format, old code may no longer work; reversing the migration may be destructive or impossible without a backup. That has to be known before the button is pressed.
- Actions that discard evidence. Restarting the process loses the state you needed. Capture a bounded diagnostic sample if it will not delay recovery or worsen pressure. A heap dump can pause the process, consume substantial disk and expose sensitive data; it is not a mandatory step before restarting.
- You cannot tell what is broken. If the mitigation would be a guess with real blast radius, spend two minutes narrowing rather than acting blindly.
Why interviews test this specifically
A live troubleshooting exercise can reveal how a candidate balances diagnosis with user impact. A candidate who narrates "the failure started at 14:02 and the deploy was at 14:00, I have checked rollback compatibility, so I am restoring the previous version and will investigate the cause afterwards" has demonstrated more than one who correctly identifies a null pointer twenty minutes in.
The related tell is silence. The round scores narration, so thinking quietly for thirty seconds reads as being stuck even when it is not.
The habit that makes it possible
Mitigations have to be fast and safe before they are useful, which means they are built in advance, not found during the incident:
- rollback that is one command and does not depend on a rebuild
- feature flags on anything risky, with a documented owner
- a load-shedding path that has been exercised
- a failover that has been rehearsed rather than diagrammed
An organisation that cannot roll back in minutes will diagnose first every time, because it has no other option. That is the real reason this principle is about engineering rather than about discipline.
Self-check
A release dropped a column the old binary reads. Is a binary rollback a safe first action? Not without a compatible data plan. Consider stopping the harmful writer or a tested forward mitigation while preserving recoverable data.