Prefill and decode are two different workloads on one GPU

TL;DR: Prefill can be compute-bound at large prompt or batch sizes. Decode emits one token per sequence per step and is often memory-bandwidth-bound at low batch sizes, because each pass reads the entire model to produce a single token. They compete on the same device, they scale on different signals, and measuring them together hides both.

Phase one: prefill

The model processes every token of the prompt in a single pass, in parallel, which makes this a large matrix multiplication that uses the tensor cores the way the hardware was designed to be used. Cost grows with prompt length. A 30,000-token prompt is not a slightly bigger version of a 300-token one; tokenwise work grows roughly a hundredfold while full-attention work has a quadratic component. Wall time also depends on kernels, caching and batch shape.

Prefill produces the KV cache entries for the prompt, plus the first output token. The user experiences it as the wait before anything appears.

Phase two: decode

Now the model generates one token at a time, and each step runs a full forward pass that reads all of the model's weights out of memory in order to produce that single token.

That is the whole reason decode behaves the way it does. At small batches, weight and KV reads can dominate the arithmetic. Larger batches amortize weight reads and can move the bottleneck toward compute; longer contexts also increase attention work and cache traffic.

Why batching helps decode so much more

If the weights must be read once per step regardless, then reading them on behalf of one sequence or of sixty costs nearly the same, and batching amortises that read across the whole batch. Throughput rises steeply with batch size. Per-sequence latency may initially rise modestly, then worsen as batch compute, cache pressure and queueing grow. Measure it at the target throughput.

A prefill workload already saturating compute may gain less from larger batches, and adding work to a compute-bound phase adds time.

This asymmetry is why continuous batching exists: sequences join and leave the running batch as they arrive and finish, instead of waiting for a fixed batch to assemble and complete together.

The interference

Both phases compete for one device, and a scheduler running one batch at a time has to choose between them. Admit the long prefill and every decoding sequence pauses while it runs, so readers watching text appear see it stop. Defer it and that user waits.

Neither choice is good, which is why chunked prefill exists: split a long prefill into pieces and interleave them with decode steps, so the worst-case interruption becomes the chunk rather than the whole prompt.

What follows for measurement

Two metrics, not one:

• Time to first token covers queue time plus prefill. This is the wait, and it scales with input length.
• Inter-token latency covers the decode loop. This is the flow, and it degrades under batch pressure.

Total response time includes first-token delay plus subsequent token gaps. Use it when the product has a completion deadline, and stratify by output length so a workload change is not mistaken for an infrastructure regression.

What follows for cost

Input tokens are processed in parallel and are cheap per token. Output tokens each cost a full pass over the weights and are expensive per token. Providers price them differently for this reason, but the bill depends on actual input/output volume, rates and cache discounts. Compare the measured cost before choosing which side to optimize.

What follows for architecture

Because the phases have different bottlenecks, they can be scaled and even hosted separately: a prefill pool chosen for compute, a decode pool chosen for memory bandwidth and cache size, with the KV cache handed between them. Disaggregation of this kind pays off at large scale and adds a distributed-systems problem to a serving stack, which is the trade to weigh rather than a default to adopt.

Before that, the cheap version of the same idea is prefix caching. When many requests share a system prompt or a document, the prefill for the shared part is computed once and reused, which removes the expensive phase entirely for that portion.

Self-check

A larger decode batch doubles throughput but pushes inter-token latency past the product target. Is the new configuration better? Only if the use case accepts that latency; otherwise reduce batch pressure or add capacity and compare cost at the same service objective.

Sources: vLLM chunked prefill tradeoffs.