The KV cache is what decides how many users you can serve
Model weights, runtime memory and the KV cache compete for GPU memory. Estimate the cache budget and memory per sequence, then check whether that concurrency also meets your latency target.
TL;DR: Reserve memory for weights, activations, runtime workspaces and allocator overhead before sizing the KV cache. For a full-attention model, cache bytes grow with stored tokens. Cache budget divided by bytes per sequence gives a memory ceiling; configured admission limits and latency can impose a lower one. A workload whose prompts get longer serves fewer users on the same hardware, with no code change and no deploy.
Two very different consumers of the same memory
Two major consumers of GPU memory are weights and the KV cache; activations and runtime buffers also need space. Weights are fixed: a 7B model in FP16 is about 14 GB and that number never moves. The KV cache is variable, and it holds, for every token of every active sequence, the key and value tensors that attention computed for it.
Weights answer "does this model fit". The cache answers "how many people can use it at once". The cache changes with the active workload, so a traffic-mix change can cause memory pressure even when the model weights are unchanged.
Why the cache exists at all
Generating token 500 requires attending over the previous 499. Caching avoids recomputing previous keys and values. Each new query still attends over stored tokens: for full attention, that part of decode work grows with context length. The cost per generated token is not constant.
The cost of that speedup is memory that grows linearly with every token, for every sequence in flight, for a full-attention cache. Sliding-window attention, eviction and offloading change the retained-memory behavior.
The arithmetic
Per token, per sequence, the cache holds a key and a value tensor for every layer:
bytes per token = 2 (K and V) × layers × kv_heads × head_dim × bytes_per_element
For an illustrative model with 32 layers, 32 KV heads, head dimension 128 and FP16 cache elements, this is 524,288 bytes, or 0.5 MiB per token. At 4,096 tokens, one sequence needs 2 GiB. A measured 60 GiB cache pool therefore fits at most 30 such sequences before allocation rounding and safety margin. This is not a universal 7B-model estimate: using eight KV heads instead of 32 cuts this cache calculation by four.
The pool size must come from the serving configuration after other memory reservations. Validate the calculation against the actual model and engine.
The consequences that surprise people
Memory bounds concurrency alongside configured admission and latency. A high sequence limit does not guarantee useful throughput: an engine may queue or preempt work under KV pressure. Check its actual admission behavior rather than assuming a high setting necessarily causes a process failure.
Longer prompts reduce capacity with no deploy. If a change doubles stored tokens per sequence, the simple full-attention estimate doubles cache bytes and halves the memory ceiling. Nothing in your change log explains the incident.
Utilisation lies here. The GPU can show high utilisation while the true constraint is cache memory. Compute is busy; admission is blocked. This is why the metric to watch is cache utilisation together with the count of requests waiting.
Eviction is a product decision. When the cache fills, an engine either queues new requests or preempts a running sequence and recomputes it later. The first adds latency at the front, the second wastes work already done. Which is right depends on whether your users are waiting on a stream.
What the design levers actually do
- Paged attention stores the cache in fixed-size blocks rather than one contiguous reservation per sequence, so memory is allocated as a sequence grows instead of at its theoretical maximum. This reduces waste from over-reservation and fragmentation, though partially filled blocks still consume memory.
- Grouped-query attention shares key and value heads across query heads, cutting cache size by a large factor. It is a model architecture choice, so it is decided before you deploy.
- Quantising the cache can reduce element storage, for example from 16 to eight bits. Measure quality and latency, and account for scale metadata and engine support.
- Prefix caching shares the cached prefix of a common system prompt or document across requests, which can avoid duplicated cache blocks and repeated prefill work for a matching resident prefix. It does not remove decode work over that prefix.
- Shorter contexts, which is unglamorous and frequently the largest available win.
The one number to carry into a room
Cache-pool bytes divided by cache bytes per sequence is a useful memory ceiling for the stated architecture. Test the resulting concurrency against latency, throughput and the prompt/output-length distribution before treating it as serving capacity.
Self-check
With the example above, what changes if the model uses eight KV heads? Cache bytes per sequence fall to 0.5 GiB, so the same 60 GiB pool has a theoretical ceiling of 120 sequences. That does not establish that 120 concurrent requests meet the latency target.