Tokens are the unit of cost, latency and capacity at once

TL;DR: Two requests per second can differ tenfold in cost and duration depending on their token counts. Instrument tokens in and out per request, tagged by feature and tenant, before optimising anything. Compute the bill from input, output and cached-token rates; generation length also affects response duration.

Why request count stops working

Web request costs vary too: a cached page and a complex search may have very different resource demands. Token counts add a useful workload measure for language models, alongside request count and shape.

One request sends 200 tokens and generates 50. The next sends 30,000 and generates 2,000. They are the same line in an access log and two orders of magnitude apart in every dimension that matters. Any dashboard, autoscaler, rate limiter or capacity model built on request count is measuring something adjacent to load rather than load.

The three things tokens decide

Cost. Providers price input and output separately, output typically several times higher per token. Self-hosting converts this into GPU-hours, but the unit that divides the bill is still tokens.

Latency. Input tokens are processed in parallel during prefill, so they mostly affect time to first token. Output tokens are produced one at a time, so they set the total duration almost linearly. If per-token decode time is unchanged, 2,000 output tokens require roughly twenty times the decode duration of 100; total latency also includes prefill and queueing.

Capacity. Every token of every active sequence occupies KV cache memory, and that memory is what limits concurrency. Longer conversations mean fewer of them at once.

One unit, three consequences, which is why these stop being separate conversations.

The asymmetry people get backwards

Use the selected model's prices. For illustrative rates of $1 per million input tokens and $10 per million output tokens, 4,000 input tokens cost $0.004 while 1,000 output tokens cost $0.01. More input tokens does not imply a larger input bill.

The practical consequence is that teams optimise the visible thing (capping maximum output length, which users notice) and leave a 4,000-token context untouched. Measure both before choosing.

Where the multiplication happens

Token spend has no natural ceiling, and it compounds in specific places:

• Retries. A retry adds another attempt; token lengths, caching and where the first attempt failed determine the extra charge.
• Agent loops. A loop that runs until it succeeds has no bound at all. A maximum step count is a cost control before it is a safety one.
• Chains. Five sequential calls means five prompts, five contexts, five latency draws and five chances to fail.
• History. Sending the full conversation every turn makes cost grow quadratically over a long session.
• Retrieval. With equal-length documents, the retrieved portion grows by 10/3. Fixed instructions and history mean total input grows by a smaller factor.

The number worth computing

Total spend is unmanageable. Cost per unit of business value (per resolved ticket, per document processed, per summary generated) is a number a team can act on and a finance conversation can use. It also answers the question that matters when a bill grows: eight times the spend alongside eight times the volume has flat unit cost, which still needs quality and revenue checks, and eight times the spend on flat volume is a defect.

Instrument before optimising

Tokens in and out, per request, tagged by feature, tenant, model and call site, with chain steps counted individually. Without that, the only available lever is turning a feature off, which is the decision a team is forced into when a bill arrives that nobody can divide.

Self-check

At the illustrative rates above, which saves more: removing 1,000 uncached input tokens or 200 output tokens? The input saving is $0.001; the output saving is $0.002. Check quality before applying either cut.