TL;DR: The average is telling you most requests are fine, which is true and useless. A gap that size means a subset of requests take a different path, so you need per-request evidence rather than aggregates: traces with exemplars, grouped by the dimension that separates the slow ones. The usual causes are queueing, garbage collection, cold caches, an unlucky shard, and retries.
How to approach it
State immediately that averages cannot answer this, then say what evidence would. The round rewards someone who reaches for traces rather than more dashboards.
A strong answer
An average over a bimodal distribution describes neither mode. If 99 percent of requests take 150ms and 1 percent take 3s, the mean is around 180ms and looks healthy. So the first move is to stop looking at aggregates and find actual slow requests. Exemplars are the mechanism: they can attach a trace ID to a histogram observation. Follow an exemplar from a slow bucket, then confirm the trace was sampled and is still retained; a p99 estimate is not itself a bucket.
Then group the slow requests and look for what they share. The candidates, in the order I would check them:
Queueing. The most common cause and the one people miss, because the service is not saturated on average. If arrivals are bursty, a queue forms during the burst and the requests behind it wait. Utilisation of seventy percent on average can still mean momentary saturation many times a minute. The evidence is that slow requests cluster in time rather than by user or endpoint.
Garbage collection or runtime pauses. A stop-the-world pause of a few hundred milliseconds hits whatever requests are in flight. The tell is that slow requests are uncorrelated with anything about the request itself, and that they cluster on one instance at a time.
Cache misses. A warm cache serves in single-digit milliseconds and a miss goes to the origin. At a 99 percent hit rate, p99 lies near the hit/miss boundary and depends on the observed distribution and quantile estimator; slower percentiles are likely to include misses. Compare slow-request latency to your known cold-path cost; if they match, you have your answer.
An unlucky backend. One slow shard, one degraded node, one replica with a failing disk. Group by instance and by shard key. This is where a p99 problem is really a single-host problem wearing a costume.
Retries and timeouts. A request that fails and retries takes the timeout plus the retry. If your client timeout is 1s with two retries, 3s is not a mystery, it is arithmetic. Check whether the slow requests are actually successful second or third attempts.
The framing worth offering at the end: tail latency is a structural property, not a tuning problem. Fan-out makes it worse mechanically, because a request that waits on twenty backends is as slow as the slowest of twenty samples from the tail. That is why hedged requests and per-try timeouts exist.
What interviewers probe next
"Which percentile should you alert on?" Something users feel, so p99 or p99.9 for user-facing paths, and always alongside a rate so you know how many people that is.
"How do you find the slow requests if you sample traces at one percent?" Tail-based sampling, which decides after the trace completes and keeps the slow ones. Random head sampling gives slow traces the same inclusion probability as other traces, but can miss rare failures in a small sample. Tail sampling needs enough buffer capacity and complete trace routing to retain the desired cases.
"Averages hid this. What else do they hide?" Bimodal anything: two datacentres, cache hit and miss, retried and first-attempt.
Common mistakes
Proposing to scale out. If the cause is GC or one bad host, scaling alone does not explain the fault, although reducing per-instance load can help some GC or queueing cases.
Looking only at server-side latency. The client sees queueing and connection setup that server metrics never record.
Treating p99 as a tuning target without asking how many requests that is. One percent of a million is ten thousand unhappy people.
References