Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.
an account raises the per-topic limit · no card
← AI & GPU Infrastructure / 12
hardNewDatabricksNVIDIAMeta
How should you adapt web-service autoscaling for an LLM inference service?
Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.
Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.
UP NEXT ON YOUR JOURNEY
Next in this trackHow is rolling out a new model different from rolling out new code, and what does a model canary actually measure?Next in this trackYou have eight 80GB GPUs and a 70B model. Tensor parallel across all eight, or two replicas of four? Defend the choice.Next in this trackOne customer's batch job is making your interactive chat slow. Fix it at the serving layer.
DISCUSSION · 0
Nothing here yet. Say how you would answer it.