Start with a service map and an evidence ledger

TL;DR: Draw the path of one failing request, then attach evidence to each boundary before changing the system.

Where you are. Start here if you can use a terminal but have not owned a production service. This lesson establishes the investigation method used throughout the course.

Give the request an identity

Our teaching service, Parcel, accepts delivery bookings over HTTPS and stores them in a database. A worker later assigns a courier. This is a fictional exercise system; its measurements are supplied fixtures. Begin with the user action: a booking request returned an error at 10:04. Record the route, timestamp with time zone, request identifier and whether the user retried. A dashboard showing average CPU cannot answer whether that booking committed.

Draw only the components involved in this operation. DNS returns an address. A load balancer terminates or forwards TLS. An application checks identity and writes a booking. A queue carries later work. Label synchronous calls separately from asynchronous processing, because a slow courier worker should not automatically make the booking API unavailable.

Separate observation from explanation

Use an evidence ledger with columns for observation, scope, hypothesis and next check. “Three requests returned 502 from one proxy” is an observation. “The database is down” is a hypothesis. The proxy might have no healthy upstream, a connection timeout, or an upstream that closed the socket. One status code does not choose among them.

In this fixture, the health endpoint succeeds while booking requests fail. That narrows the problem only if you know what health checks. A process that answers a static health response may still be unable to write to storage. Compare a failing booking with a successful booking in the same interval; checking a different environment would introduce another variable.

Prefer a check whose possible outcomes lead to different actions. Reading the proxy's upstream error and the application's request log is more discriminating than restarting every component. Preserve the relevant log window first. If the service is causing material user harm, a bounded mitigation can happen while another responder investigates; investigation order is not a reason to delay recovery.

Explain your next move aloud

A useful interview response states the observation, the remaining alternatives and what the next measurement will separate. For this exercise: “The proxy answered, but the booking route failed. I will compare upstream connection errors with application request IDs to determine whether requests reached the application.” That answer exposes your reasoning to follow-up questions.

The method follows the distinction between symptoms and underlying causes in Google SRE monitoring guidance. The ledger and Parcel scenario are practice devices. They do not claim to reproduce a named company's interview.

Compare the evidence

Do this before moving on

Create a five-row ledger for these facts: DNS resolves; TLS succeeds; the proxy returns 502; the health route returns 200; no booking request ID appears in application logs. Choose one next check and state both possible outcomes.

Expected reasoning: inspect proxy upstream connection details and log completeness before blaming storage. Missing application logs could mean the request never arrived or logging failed. Give yourself one point for identifying that ambiguity, one for a scoped check, and one for avoiding an unsupported database diagnosis. Rework the ledger if you score below three.

Go deeper

• Diagnostic method explains how evidence changes the next investigation step.
• Queues prepares you to identify where waiting occurs.
• Cascading failures extends the map when retries spread the incident.

Key takeaways

• Follow one operation across its actual dependencies.
• A missing log is evidence with a collection caveat.
• Choose measurements that separate competing explanations.

Check yourself

Say it before you reveal it. Pulling the answer out of your own head is what fixes it. Nodding along to someone else's does nothing.

1. 1The health route succeeds. What follows?

   All dependencies workThe health endpoint answered its own checkThe database is healthy

2. 2What belongs in the first ledger row?

   Answer it, then reveal