DevOpsInterviewPrep logo
Observability, SLOs & Reliability / 02
easyNewDatadogAtlassianAccenture

Logs, metrics and traces: what is each one good at, and which do you reach for first in an incident?

Metrics tell you something is wrong, traces tell you where, logs tell you why. Answering in that order is also the order you should open them at 3am, and it is what stops an incident starting in a log search.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Metrics are cheap aggregates that detect and alert. Traces follow one request across services and localise the problem. Logs are expensive detail that explains it. Start at metrics, narrow with traces, confirm in logs, and never start an incident by grepping.

How to approach it

Give each one a job and a cost, then give the order you use them in. The order is the part that shows operational experience, because the common failure is someone opening a log search first and reading thousands of lines with no idea which service is at fault.

A strong answer

Metrics are numbers aggregated over time: request rate, error rate, latency percentiles, queue depth, memory. Storage cost is tiny because a counter is one number per series per scrape regardless of how much traffic it counted. That makes them the right thing to alert on and the right thing to keep for a year. What they cannot tell you is anything about an individual request, and their resolution is limited by cardinality: you can break latency down by endpoint, not by customer.

Traces follow one request through every service it touched. A trace is a tree of spans, each with a start, a duration and attributes, tied together by a trace ID propagated in headers. A trace answers the question metrics cannot: of the 900ms this request took, where did it go. This is the only tool that gives you that across a distributed system, and it is why "latency is up somewhere" stops being a hunt. Cost is managed by sampling, since storing every trace at high volume is not affordable, and the sampling strategy is the interesting design decision: head-based is cheap and may miss the rare slow request, tail-based keeps the interesting ones and requires buffering.

Logs are the detailed record: this specific thing happened, with this context, at this moment. They carry the stack trace, the parameter values, the exact error from the database. That is the highest-value information per line and the highest cost per byte, because every line is stored and indexed. Logging is usually the largest line on an observability bill, and frequently larger than the database it is monitoring.

The order in an incident:

An alert fires on a metric, because that is what alerts are built from. Error rate above threshold, or latency burning the error budget. That tells you something is wrong and roughly where.

Open a trace for a failing or slow request. That tells you which service, and which call inside it, is responsible. This is the step people skip and it is the one that saves the most time, because without it you are guessing which service's logs to open.

Read the logs for that service, in that time window, ideally filtered by the trace ID. Now you are reading twenty relevant lines instead of searching a million.

That last point is the practical glue: inject the trace ID into every log line. Then a trace links directly to its logs and the three stop being three separate tools. OpenTelemetry exists largely to make that correlation standard across languages and vendors rather than something each team wires up differently.

What to instrument on day one: the four golden signals as metrics (traffic, errors, latency, saturation), tracing on every inbound and outbound call, and structured logs as JSON with a trace ID and a level, at info for lifecycle events and error for things a human should see.

What interviewers probe next

"Where do events and profiles fit?" Events are discrete records of something notable, a deploy or a config change, and overlaying them on a metric graph answers "what changed" faster than anything else. Continuous profiling adds a fourth signal for CPU and memory inside a process, which traces cannot see.

"How do you cut the logging bill without losing the ability to debug?" Drop the lines nobody reads, which is usually most of the debug and info volume from healthy requests, and keep everything for errors. Sampling successful requests while keeping all failures preserves nearly all the diagnostic value.

"Can you get metrics from logs?" Yes, by counting matching lines, and it is expensive and slow compared with a counter. Useful for something you failed to instrument, not a design.

Common mistakes

Opening a log search first, which is the slowest path to the answer and the one most people take.

Alerting on log patterns rather than metrics, which is noisy and breaks whenever a message string changes.

Logging without a trace ID or a request ID, so correlating anything means guessing by timestamp.

Treating traces as a debugging luxury, then having no way to localise latency across six services.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.