inference
DevOps interview questions tagged inference, across every topic.
3 questions · 1 unlocked for you
Concepts behind "inference"
The concept pages behind what these questions are testing.
Foundational
The KV cache is what decides how many users you can serveModel weights, runtime memory and the KV cache compete for GPU memory. Estimate the cache budget and memory per sequence, then check whether that concurrency also meets your latency target.🤖 AI Infrastructure
Foundational
Prefill and decode are two different workloads on one GPUPrefill builds prompt state; decode extends each sequence token by token. Their bottlenecks depend on model, batch size and context length. Measure first-token and inter-token latency separately before choosing batching or disaggregation.🤖 AI Infrastructure
Core
Why GPUs break the assumptions schedulers are built onGPU scheduling must account for device memory, sharing boundaries and interconnect topology. Device plugins, MIG, DRA and gang scheduling address different constraints; none makes accelerators behave like interchangeable CPU millicores.🤖 AI InfrastructureSign in