tracing

tcpdump copies packets to userland; eBPF aggregates inside the kernel next to the event. Know which hook runs where, what a verifier log is telling you, and when XDP drops for free.

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

Day-one telemetry is cheap and retrofitting it costs an incident. The answer that scores names the linking layer (trace IDs in logs, exemplars on metrics) that turns three tools into one story.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

Assert on the shape of a trace: which spans ran, in what order, with which attributes. Catches the cross-service regressions unit and contract tests structurally miss, at the price of instrumentation discipline.

There is no status code and no caller waiting, so latency and error rate stop meaning what they meant. The signals that replace them are age, lag and completion, and they answer a question the request-response set never had to ask.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.