opentelemetry

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

OpenTelemetry is an API and a wire protocol, not a product. Candidates who promise instant multi-vendor bliss have never run the collector in production or negotiated with a vendor who knows it.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

Assert on the shape of a trace: which spans ran, in what order, with which attributes. Catches the cross-service regressions unit and contract tests structurally miss, at the price of instrumentation discipline.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

HTTP success can carry a rejected-span count. Replaying that batch can duplicate accepted telemetry, while enlarging the queue cannot repair invalid data.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.