DevOpsInterviewPrep logo
Observability, SLOs & Reliability / 06
mediumNewDatadogUberStripe

A request crosses six services and you have no idea where the time goes. How does tracing fix that?

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: A trace ID is created at the entry point and propagated in headers to every downstream call, so each service records a span linked to the same trace. The critical property is that propagation must be unbroken: one service that does not forward the header splits the trace in two, and the gap is invisible rather than reported as an error.

How to approach it

Explain the mechanism concretely (headers, IDs, parent links) rather than describing what a trace UI looks like. Then get to sampling, which is where the practical decisions are.

A strong answer

A trace is a tree of spans. When a request arrives at the edge with no trace context, the entry service generates a trace ID and a root span ID. Every outbound call carries them in a header, standardised by W3C Trace Context as traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ^  ^                                ^                ^
          version  trace-id                   parent-span-id   flags

The downstream service reads it, creates a span whose parent is that span ID, and forwards its own ID onward. The collector reassembles the tree by trace ID, and the result answers the question that metrics cannot: for this specific request, how long did each hop take and which one was slow.

The failure mode is a broken chain. One service that does not propagate the header, or a queue or async boundary where context is dropped, produces two disconnected traces instead of one. Nothing errors; the trace simply ends and a new one begins. Instrument critical boundaries and make known gaps visible; partial traces still provide useful evidence if their limits are understood.

Sampling is the practical decision. Tracing every request at volume is expensive in ingest and storage, so you sample. Head-based sampling decides at the entry point, typically keeping a small percentage, which is cheap. Random head sampling is unbiased with respect to outcomes but can miss rare failures. Tail-based sampling buffers candidate spans until a decision deadline, enabling error/latency policies plus a normal baseline. It cannot recover upstream drops, and late spans or capacity limits can leave incomplete traces. Tail-based is what you want and it costs more, because the collector must hold traces in flight.

Exemplars are the bridge to metrics. A histogram bucket can carry a trace ID for one of the observations in it, so from a latency graph showing a bad p99 you can jump directly to a trace of a request in that bucket. That link is what makes metrics and traces one workflow instead of two tools, and it is the answer to "how do I find a slow request when I only sample one percent".

Span attributes can carry request detail with explicit cost and privacy controls. Identifiers and full URLs can create metric-series growth. Use only necessary span attributes, redact sensitive data, and budget indexing and retention; a trace is not a safe place for arbitrary personal data. Saying that in the interview shows you understand why the three signals exist separately.

What interviewers probe next

"How do you propagate across a queue?" Put the trace context in the message metadata and choose parent relationships or span links according to the messaging semantic convention and processing model; batching and multiple causal inputs often need links.

"What does instrumentation cost?" Auto-instrumentation covers common frameworks with near-zero code change and some overhead. The value comes from manual spans around the parts specific to your domain.

"Traces or logs for debugging?" Traces for where the time went across services, logs for what happened inside one. Correlated by trace ID so you can pivot between them.

Common mistakes

Describing the UI rather than the propagation mechanism.

Head-based sampling at one percent, then wondering why no trace exists for the incident.

Putting high-cardinality identifiers on metrics because tracing was never adopted, which is how a cardinality explosion starts.

References

OTel sampling.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.