latency

Multiplexing removes application-level queuing and hands the problem to TCP. Here is where the blocking really lives, the loss math behind it, and the four levers that help.

Turns the standard autoscaling definition into the problem it actually causes. Almost every candidate can define an HPA; far fewer can say why theirs did not save them.

Latency budgets decide architecture, and most teams buy edge compute before earning it with caching and protocol work. Know which rung of the ladder your problem actually needs.

Fail protects the policy, Ignore protects availability, and every real answer starts with per-webhook threat model plus scoping. The canonical outage, a webhook that rejects its own restart, separates the people who have run this from those who read about it.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

Type-ahead is a latency budget wearing a search costume: keystrokes arrive every 200 milliseconds, so anything slower feels broken. Prefix indexes, freshness tiers, and typo tolerance are the scored parts.

Reciting both acronyms is the median answer. Expert level chooses by connectivity assumptions and undo semantics, then defends CRDT metadata growth with numbers.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

One request touches an embedding call, a vector search, a rerank, a prompt build and a generation, and a single duration for the whole thing tells you nothing about any of them.

The database part is easy. The operational problems are reindexing when the embedding model changes, recall degrading silently as filters get stricter, and memory growing faster than anyone forecast.