Autoscaling control loops: signals, delay and usable capacity
Understand autoscaling as delayed feedback. Work through replica calculations, startup backlog, stabilization and the dependency limits that extra workers cannot fix.
TL;DR: Autoscaling changes capacity after a delay. Choose a signal that falls when useful capacity increases, retain headroom for the delay, and check that new workers can become ready without overloading their dependencies.
A replica recommendation is only the start
An autoscaler observes a measurement, compares it with a target and requests a change. The application benefits only after the additional instance accepts useful work. Between those events sit scheduler placement, image download, initialization, readiness checks and sometimes provisioning a new node. A dashboard showing the desired replica count can hide all of that waiting.
The Kubernetes HPA's basic proportional calculation is ceil(current replicas × current metric / target metric). CPU utilization is measured relative to resource requests. Readiness, missing samples, tolerance and scaling policies can alter the final action; multiple metrics generally select the largest recommendation. The HPA algorithm documentation describes these qualifications. Treat the formula as a starting estimate, not a promise of the next observed replica count.
The loop passes through completed work. If a database lock stops every worker, more pods may create more waiting connections while request latency keeps rising. Scheduling constraints explain another limit: nodes must fit the workload's actual placement requirements.
Work the delay into the calculation
Consider a hypothetical service with four replicas, identical requests and evenly distributed CPU demand. Average CPU utilization is 90% against a 60% target. Ignoring damping and limits, the calculation requests ceil(4 × 90 / 60) = 6 replicas.
Now use a separate load-test observation: one ready replica sustainably completes 100 requests per second at the required latency. Traffic rises from 350 to 550 requests per second. Four replicas can serve about 400, so the queue grows by approximately 150 requests per second while additional capacity starts. If detection plus startup takes 80 seconds, the simple fluid estimate adds 150 × 80 = 12,000 waiting requests. This assumes constant arrivals, no abandonment and no load shedding; a real request deadline may expire well before the queue reaches that size.
Six replicas provide 600 requests per second, leaving only 50 requests per second to drain that backlog. Clearing 12,000 requests therefore takes another 240 seconds under those assumptions. “Scale to six” solves steady capacity but fails a short latency objective during recovery. Existing spare capacity, earlier scaling or bounded admission is necessary. Predictable scheduled traffic can justify pre-scaling; irregular bursts require a measured headroom policy.
Select a signal with a causal relationship
| Signal | Useful when | Misleading when |
|---|---|---|
| CPU per replica | CPU work dominates service time | Workers wait on a database or remote API |
| Queue work per ready worker | Jobs have comparable service cost | One job can represent seconds or hours |
| Request concurrency | Each replica has a measured concurrency limit | Hung requests inflate concurrency indefinitely |
| Oldest queued item age | Delay is the service contract | A poison item ages while ordinary work succeeds |
Queue depth alone lacks units of work. For an image processor, 200 thumbnails and 200 large panoramas can need very different capacity. Split workload classes or estimate remaining service time, then validate the estimate against actual completion rates. Pair any demand metric with successful throughput and latency so a controller cannot declare victory merely by moving requests into another queue.
Stabilize removal and bound growth
Downscale stabilization retains recent higher recommendations to reduce repeated removal and recreation during fluctuating demand. Scaling-rate limits bound how quickly the requested capacity changes. These controls address different problems: smoothing removal does not make startup faster, and a maximum replica count does not protect a database unless its connection and query budgets were included in that maximum.
Suppose every pod can open 20 database connections. Increasing from four to twelve pods raises the potential total from 80 to 240. If the database budget for this service is 160, an unconstrained HPA trades an application queue for a connection storm. Budget rollout surge and background workers too. Connection pool exhaustion follows that failure through both queues.
An interview follow-up is whether an HPA error should remove capacity. Missing metrics are evidence of uncertainty, not proof of low demand. Explain the controller's configured behavior, alert on persistent observation failures and avoid inventing a healthy zero.
Self-check: six workers can complete 600 jobs per second, arrivals are 600, and the queue contains 18,000 jobs. Does scaling to six drain it? No. There is no spare processing rate. At eight equally capable workers, the ideal drain time is 18,000 / (800 - 600) = 90 seconds, after the additional workers become ready and only if downstream capacity supports that rate.