batching
DevOps interview questions tagged batching, across every topic.
2 questions · 0 unlocked for you
Concepts behind "batching"
The concept pages behind what these questions are testing.
Foundational
The KV cache is what decides how many users you can serveModel weights, runtime memory and the KV cache compete for GPU memory. Estimate the cache budget and memory per sequence, then check whether that concurrency also meets your latency target.🤖 AI Infrastructure
Foundational
Prefill and decode are two different workloads on one GPUPrefill builds prompt state; decode extends each sequence token by token. Their bottlenecks depend on model, batch size and context length. Measure first-token and inter-token latency separately before choosing batching or disaggregation.🤖 AI Infrastructure