Everything is a queue, and the queue is where the latency lives
A packet queues at the NIC, a connection queues in the accept backlog, a request queues for a worker, a query queues for a connection, a write queues at the disk. Queueing is one source of latency; service time, fan-out and network delay matter too. A queueing model is useful only when its assumptions match the workload.
TL;DR: In an M/M/1 model with unchanged mean service time, 80 percent utilisation gives four times the mean queue wait of 50 percent. Real services need a model of their arrivals, worker count and service times before that curve can justify a headroom target.
The chain of queues
Follow one request and count the places it can wait. The NIC ring buffer. The kernel's accept backlog, if every worker is busy. The application's own work queue. A connection pool slot, if the database has fewer connections than the service has threads. The database's own lock queue. The disk's IO scheduler.
For sequential work, latency includes queue waits plus execution and transit time. Execution may dominate. Trace the slow request to distinguish waiting for a worker from a slow computation, lock holder or downstream call.
A single-server model
The M/M/1 model assumes Poisson arrivals, independent exponentially distributed service times, one server and an unbounded first-come-first-served queue in steady state. For arrival rate λ and service rate μ, stability requires ρ = λ/μ < 1. Mean waiting time before service is Wq = ρ / (1 − ρ) × E[S], where E[S] = 1/μ. This is a mean, not a p99. MIT’s queueing chapter derives these results.
Under those assumptions: at 50 percent utilisation a request waits about as long as it takes to serve. At 80 percent it waits about four times that. At 90 percent, nine times. At 95, nineteen.
A multiworker pool, deterministic service times or bounded queues produce different results. Derive the utilisation target from measured latency under representative load, including bursts. A fixed 60 or 70 percent target is a starting hypothesis, not a universal optimum.
Why bursts matter more than averages
The arithmetic above assumes a stationary Poisson arrival process, which already includes random arrival gaps; it does not model arbitrary correlated bursts. Real traffic arrives in bursts, and a queue that is fine on average forms during every burst and drains afterwards.
So a service averaging 40 percent utilisation can have a p99 dominated by queueing, because for a few hundred milliseconds many times a minute it is momentarily above capacity. Short overloads can disappear in fifteen-second averages. Inspect queue wait distributions and finer-resolution traffic alongside traces before attributing the tail to bursts.
Buffers absorb bursts, not sustained overload
A larger bounded buffer can absorb a finite burst if the service catches up before callers’ deadlines. Under sustained overload, it mostly increases waiting before rejection or timeout.
A request sitting in a 10,000-deep queue for eight seconds, then being served, is worse for the user than a request rejected immediately, because the client has usually given up and retried, so you are now doing the work twice for an answer nobody wants. This is bufferbloat in the network and it is the same mistake in an application.
For sustained overload, bound the queue, reject or shed when it is full, and apply backpressure so the producer slows down rather than the buffer growing. Choose queue capacity against the waiting-time budget and discard expired work before executing it.
What each response actually fixes
- More capacity lowers utilisation, which moves you back down the curve. It fixes queueing caused by genuine load.
- Faster service does the same thing from the other direction, and is usually cheaper.
- Shedding load protects the requests you keep by refusing the ones you cannot serve. It fixes nothing and prevents collapse.
- A bigger bounded buffer can absorb a short burst, but adds no service capacity and can make sustained overload fail more slowly.
Diagnosing which queue is full comes first, because these are not interchangeable. Adding replicas to a service whose requests are queueing for database connections adds more competitors for the same pool, and makes it worse.
The one habit
When latency rises, ask where the request is waiting before asking why the code is slow. Measure queue depth and queue wait alongside execution time and resource saturation; high latency alone does not establish that a queue is full.
Self-check
Can a larger queue absorb a short finite burst without increasing steady-state capacity? Yes, if the backlog drains within the latency and memory budget. It cannot make sustained arrival above service capacity stable.