DevOpsInterviewPrep logo
🚨 Debugging Production
Foundational

Diagnostic method: how to debug a system you did not build

The troubleshooting round is not testing which commands you know. It is testing whether you narrow a search space deliberately, and whether you protect users before you satisfy your curiosity. The sequence below gives you a practical structure for impact assessment, hypothesis testing and recovery.

TL;DR: Establish impact, define the symptom precisely, bound the blast radius and the timeline, then form hypotheses you can eliminate rather than confirm. Mitigate before you root-cause. The order is the skill; the commands are lookup.

Why method beats knowledge here

You will be handed a system you have never seen, with incomplete information, and watched. Nobody expects you to know the answer. They are watching whether your next action reduces uncertainty or merely produces output.

The failure mode is recognisable: a candidate lists every command they know, runs them in no particular order, and narrates results without interpretation. It reads as memorisation because it is. The alternative is to say what you believe, what would prove you wrong, and then go and look.

The sequence

Establish impact. Who is affected and how badly? This decides urgency and whether you are allowed to take disruptive action. "All users, checkout failing" and "one internal dashboard is slow" deserve different responses, and starting here stops you over-reacting to the second.

Define the symptom precisely. Not "it is slow" but "p99 on the checkout endpoint went from 200ms to 3s at 19:40, affecting roughly 40 percent of requests". Precision bounds the search immediately: something that started at a moment has a trigger, and something affecting a fraction is probably not global config.

Bound it. Is it one region, one node, one customer, one endpoint? Every dimension you can rule out removes a branch. This is the highest-value step per unit of time and the one most often skipped.

Establish the timeline. What changed near the start? Deploys, config changes, feature flags, certificate rotations, traffic events, upstream incidents. Correlation is not proof, but it is where to look first, and most incidents do have a trigger.

Form hypotheses that are falsifiable. State the belief and the observation that would kill it. "If the new release's request path is the cause, a completed rollback with compatible state should restore latency" gives you a test with explicit conditions. Persistent data changes or an incomplete rollback can leave symptoms in place. "It might be the network" is neither.

Mitigate before root cause. If a rollback, a failover, a flag flip or shedding load restores service, do it and diagnose with the pressure off. Interviewers weight this heavily because it is the difference between someone who protects users and someone who protects their own interest in the puzzle.

Then root cause, validate, and prevent. Confirm recovery is real rather than coincidental, and turn the finding into a change with an owner.

The one habit to carry into the room

Narrate. Reasoning done silently earns nothing, and long pauses read as being stuck. Say the boring sentence out loud: "I am checking whether this correlates with a deploy, because that will help prioritize change-related hypotheses; delayed effects mean a timing mismatch does not rule them out."

Self-check

Given "the service is slow", what are the first three questions you ask before touching a terminal? And if a rollback restores service but you do not yet know why, is the incident over?

RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS