DevOpsInterviewPrep logo
← AI & GPU Infrastructure / 19
expertNewNVIDIADatabricksMicrosoft

Long prompts are making short requests slow on the same replica. What is happening, and how would you fix it?

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

an account raises the per-topic limit · no card
UP NEXT ON YOUR JOURNEY
Next in this trackDownstream automation needs valid JSON every time. How do you get it, and what does it cost?Next in this trackForty customers each want a fine-tuned model. You have eight GPUs. How do you serve that?Next in this trackSpeculative decoding promises a big latency win. When does it not deliver, and what does it cost you?
DISCUSSION · 0

Nothing here yet. Say how you would answer it.