Metrics, logs and traces: designing evidence for production questions

TL;DR: Metrics summarize measurements across time and dimensions, logs record events, and traces connect operations across a request or workflow. Design them around questions you must answer during a failure. Collecting all three does not guarantee that the needed evidence exists or can be correlated.

Start with a production question

A checkout API's p95 latency rises from 250 milliseconds to 1.8 seconds after a deployment. You need to know how many users are affected, which operation slowed, where time was spent and whether the behavior belongs to the new revision. No single telemetry type answers every part efficiently.

Metrics can show the affected request fraction and timing by route and revision. Traces can reveal that slow requests spent most of their time waiting for the inventory service. Structured logs can explain that inventory calls were retrying after a configuration change. The times and values here are an illustrative investigation, not reported production measurements.

OpenTelemetry's signal overview defines telemetry signals and their roles. OpenTelemetry instrumentation and collection help produce and transport evidence; you still need a backend, retention policy and a useful operational model.

Choose the evidence by its job

Keep IDs with unbounded variety, such as request IDs, out of ordinary metric label sets. They fit better in logs and trace context, subject to privacy and retention policy. Use bounded route templates such as /orders/{id} instead of raw paths containing every order number.

Correlation needs consistent fields

Carry service identity, environment and deployment revision consistently across telemetry. Propagate trace context through supported HTTP clients and messaging boundaries. Include trace and span identifiers in relevant structured logs so an investigator can move from a representative slow request to its events.

For asynchronous work, the producer and consumer may execute at different times with different parentage. Use the instrumentation model's supported links or context propagation rather than inventing a single synchronous span that lasts through hours of queue delay. Measure queue waiting time separately from processing time.

Time synchronization matters when comparing hosts, but a timestamp alone is a fragile join key. Two services can emit thousands of events in the same second. A meaningful operation ID or propagated trace context makes the relationship explicit.

Sampling changes what absence means

A sampled trace backend may contain no example of a rare error even though metrics clearly show it happened. That absence cannot prove the error path is healthy. Head sampling chooses earlier in the request, while tail sampling can make a decision after observing more of the trace but needs collection capacity and a decision window.

Decide what must remain measurable independently of trace sampling. Customer request and error counts typically need reliable aggregate instrumentation. For sensitive operations, retain carefully scoped diagnostic events under an explicit policy rather than indiscriminately increasing payload capture.

The OpenTelemetry sampling documentation explains the tradeoffs. Test the behavior of your actual collector pipeline, including what happens when buffers fill or a telemetry backend becomes unavailable. A monitoring system can silently lose the very evidence needed during overload unless you monitor its own delivery health.

Alert on a condition with an action

A dashboard supports exploration. An alert interrupts someone because a condition requires timely action. For the checkout example, a sustained customer error or latency objective violation can justify a page. An individual Pod restarting successfully may be useful diagnostic evidence without requiring an immediate human response.

The Google SRE monitoring chapter distinguishes monitoring signals and the cost of noisy alerting. Connect every page to an owner, a clear symptom and a first diagnostic or mitigation step. Route slower capacity trends to planned work when their time-to-impact allows it.

Keep monitoring and observability related but distinct in your explanation. Monitoring checks known conditions; well-designed telemetry also lets engineers investigate unexpected behavior. That broader capability depends on instrumentation choices and the ability to ask new questions of retained data.

Grafana dashboard design turns those signals into an incident view with traffic-weighted error ratios, scoped drill-downs and reviewed provisioning files.

Reconstruct the checkout incident

Self-check: aggregate latency is normal, but customers in one region report slow checkouts. Traces show no errors. What could you be missing?

Segment latency and success by a bounded region label and relevant operation. Check the sampled population, trace propagation and whether “success” includes a timed-out client after the server eventually returns 200. Inspect network and client-side evidence where server telemetry cannot observe the failure. Compare revision and traffic mix before attributing it to the deployment.

Practice logs, metrics and traces, then write a specific paging condition using symptom-based alerting and error-budget burn rates.

Follow those signals through OpenTelemetry Collector architecture, including processor order, exporter queues and the loss window during a restart.