observability

Every ops engineer schedules jobs; few can say which shell cron uses, where stdout lands, or why percent signs delete half a command. Those gaps are exactly what interviews test.

Everything top, htop and your APM agent display is derived from procfs anyway. Knowing the raw files turns any bare SSH session into a monitoring console.

tcpdump copies packets to userland; eBPF aggregates inside the kernel next to the event. Know which hook runs where, what a verifier log is telling you, and when XDP drops for free.

Asked in nearly every senior Kubernetes loop, and most candidates answer with a feature list. The scoring answer names the three problems it solves and then argues for not adopting it.

The loop is easy; the statistics are not. This question finds engineers who can size a traffic slice against sample noise and name the false rollback that kills trust in automation.

Kubelet resource metrics to metrics-server to aggregated API to HPA controller: every hop adds lag and averaging. Walking the path and naming where it distorts demand separates operators from dashboard readers.

Exit codes, the previous-container log, and a little-known file at /dev/termination-log that carries a container's last words into kubectl output. Scale triage starts with classification, not with opening pods.

Best-effort objects, one-hour TTL, deduplicated by count, invisible unless shipped out. Events narrate scheduling and lifecycle decisions; treating them as durable records is the expensive mistake.

The observability failure that scales with your success. Cardinality is the cost driver in every metrics system, and the fix has to be structural or it comes back next quarter.

A question with a right answer and a subtle reason. Histogram versus summary is the part that catches people, and it turns on whether you can aggregate across instances.

Tracing is the only telemetry that answers where the time went across services. The mechanism is context propagation, and the reason teams fail at it is that one uninstrumented hop breaks the chain.

The instinct is to delete old logs. That is backwards: the waste lives in what you ingest, not how long you keep it. This answer walks the levers in the order that actually pays.

OpenTelemetry is an API and a wire protocol, not a product. Candidates who promise instant multi-vendor bliss have never run the collector in production or negotiated with a vendor who knows it.

Day-one telemetry is cheap and retrofitting it costs an incident. The answer that scores names the linking layer (trace IDs in logs, exemplars on metrics) that turns three tools into one story.

Choose what evidence must survive, then price the collection path. A tail policy can only inspect traces that reach it; low SDK sampling changes the guarantee before the collector sees a span.

The kernel watches syscalls, sockets and short-lived processes that userspace agents polling every few seconds never meet. Know what eBPF adds, what it costs, and the security conversation it starts.

One surface cannot serve the responder and the review. Tier dashboards by the question each audience asks, and make every tile answer in five seconds or link one click deeper.

You have added a dependency with a latency distribution an order of magnitude wider than anything else in your stack, whose failure mode is a plausible wrong answer rather than an error.

Agent per node, gateway per cluster, and one decision that shapes everything else: tail sampling needs every span of a trace in one place, which is why the topology has a tier that looks redundant until you need it.

Tracing tells you which service and which span. It stops at the function boundary, and that is exactly where a CPU regression lives. Continuous profiling is the layer most teams do not have when they need it.

There is no status code and no caller waiting, so latency and error rate stop meaning what they meant. The signals that replace them are age, lag and completion, and they answer a question the request-response set never had to ask.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

Exit codes said green while the warehouse went hollow. Where detection should have sat, why retries could not help, and the backfill discipline finance will actually accept.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Everyone sizes a metrics system by samples per second and everyone is wrong. The constraint that actually kills these platforms is a single label added on a Friday.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

Prompts contain whatever the user typed, which in a support product is names, account numbers and medical detail. Full-text logging is a data-protection decision, not a debugging preference.

Latency is flat, errors are zero, and the answers are worse. Quality regression is invisible to every metric a platform team normally owns, which is why it needs its own pipeline.