OpenTelemetry Collector architecture: follow telemetry, queues and failure boundaries
Explain OpenTelemetry Collector receivers, processors, exporters and agent/gateway roles. Work through queue exhaustion and validate a local trace pipeline.
TL;DR: The Collector receives telemetry, applies ordered processing and exports it through configured pipelines. Its queues absorb bounded disruption; verify what acknowledgement, restart and downstream failure mean at each hop before promising data retention.
Configuration defines a graph of consumers
Receivers accept or collect a signal. Processors modify, batch, sample or reject it. Exporters deliver it to a backend or another Collector. Defining a component alone does not enable it: the service.pipelines configuration must reference it, and the components must support that pipeline's signal type.
The Collector architecture describes receiver fan-out and ordered processors. A receiver referenced by several pipelines can feed them from one instance; processor instances are separate per pipeline even when they share configuration. A synchronous blocking consumer can affect other paths sharing that receiver. Drawing two arrows does not establish independent failure isolation.
This is a fan-out example, not automatic failover. Both exporters receive data. Their error handling and buffers need verification before one backend is treated as a fallback for the other.
Start with a local observable path
Save this as collector.yaml for an OpenTelemetry Collector distribution containing otlp, memory_limiter, batch and debug. It binds OTLP/HTTP to loopback and exports a basic summary locally. The example does not configure production transport or persistent delivery.
receivers:
otlp:
protocols:
http:
endpoint: 127.0.0.1:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 128
spike_limit_mib: 32
batch:
timeout: 1s
send_batch_size: 128
exporters:
debug:
verbosity: basic
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [debug]
Validate the file with the distribution's validate --config=collector.yaml command, then start it with that config and send a synthetic trace from a local OTLP client. Expect a trace summary, not a dashboard. Remove the debug reference from the pipeline to see why a defined exporter is not enough; validation should reject a pipeline without an exporter rather than magically selecting one.
The configuration guide explains activation and component IDs. The memory limiter should be early enough to signal pressure before additional processing allocations; it is not an operating-system memory limit or a guarantee that the process cannot OOM. The values here are illustrative and require measurement for a real Collector.
| Location | Evidence to observe | Failure it helps distinguish |
|---|---|---|
| Receiver | Accepted and refused data | Client delivery versus admission failure |
| Processor | Drops, errors and memory pressure | Intentional filtering versus overload |
| Exporter queue | Occupancy, capacity and failed enqueue | Bounded buffering versus sustained backlog |
| Backend | Accepted and queryable data | Export attempts versus usable stored telemetry |
Calculate how long a queue can help
Assume an illustrative exporter queue can hold 10,000 equally sized requests and receives 200 requests per second while the backend is unavailable. Starting empty, it has about 50 seconds of capacity. Real queue sizing may use items, bytes or requests depending on the component/version; use the configured unit and actual batch-size distribution.
After a 20-second outage, about 4,000 requests await export. If the recovered backend accepts 300 requests per second while arrivals remain 200, the net drain is 100 per second and recovery takes another 40 seconds. A backend that can only match the arrival rate never drains the backlog. Extra retries do not create throughput.
Collector resiliency guidance describes sending queues, retry bounds and file-backed persistence. An in-memory queue loses its contents on a process crash. A persistent queue can survive supported restarts, but disk failure, full storage and retry exhaustion still matter. Neither path implies exactly-once ingestion: ambiguous acknowledgements can cause repeated delivery, and backend behavior determines its consequence.
Choose agents and gateways for a reason
A node-local agent is useful for host context and local collection. A gateway centralizes export credentials and expensive processing. It also becomes a shared capacity dependency. Tail sampling adds a further constraint: spans for one trace must reach the stateful decision point consistently, with enough time and memory to make the sampling decision.
For an interview, trace one missing span through a named receiver, pipeline and exporter. “The Collector is healthy” only establishes the process survived. Compare accepted input with filtered, refused and successfully exported data over aligned windows, accounting for batching and queue delay.
Head and tail sampling adds trace-routing requirements, a memory budget and selection bias to this collection path.
Self-check: the receiver accepted a trace, the Collector restarted before export, and only an in-memory queue was configured. Was the trace durably stored? No. Receiver acceptance did not establish backend persistence. Decide which loss window the service can tolerate, then test that boundary with synthetic telemetry and a controlled interruption.
A working collection path still needs meaningful relationships between spans. Tracing across queues covers message carriers, batch links and separate retry attempts.