NVIDIA AI & GPU Infrastructure interview questions

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Three sharing modes with three different isolation guarantees. MIG partitions the hardware, MPS shares the SMs, time-slicing just interleaves contexts, and picking wrong turns a latency SLO into a lottery.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.

GPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.

The driver, the container toolkit, the CUDA runtime inside the image and the framework build all have to agree. Upgrade the wrong one first and every pod on the node fails to start with an error that names none of them.

At sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

The fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.

CPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.

Requests-per-second is not the load unit, cold starts are minutes not seconds, and scale-in can kill paying users mid-sentence. Everything you know about HPA needs re-deriving here.

Both configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.

Separate scheduling delay from image pull, artifact transfer, GPU loading and warm-up. The autoscaler needs the full request-to-ready time.

The infrastructure win is real and easy to measure. The quality regression is real, harder to measure, and will not show up in your latency dashboards at all.

It reads as a configuration change and it is a capacity change. Cache memory per request grows with the limit, prefill cost grows with what people actually send, and concurrency falls in proportion.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.

Compatible LoRA adapters let tenants share resident base weights. Check adapter size, KV-cache demand and traffic before sizing the replica pool.

It trades compute for latency, so it wins on an underloaded fleet and can lose on a saturated one. The acceptance rate decides everything, and the acceptance rate depends on traffic you do not control.

A request that streams for ninety seconds is not slow, it is long. Total latency mixes the part users wait for with the part they read, and an SLO built on it will page you for success.

The utilisation number everyone quotes means at least one kernel was resident, not that the hardware was doing useful work. A GPU at 95 percent can be doing five percent of what it is capable of.

A training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Code and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.

Failure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.

Expensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.

A notebook holds a GPU while somebody thinks, which is most of the day. The utilisation on an interactive fleet is the worst in the building, and the fix is a queue people do not hate.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

GPU capacity is not elastic the way CPU is. On-demand means whatever is left after the reservations, and in a tight region that is nothing, so the availability problem and the cost problem are the same problem.

Code rollouts ask 'does it work'; model rollouts ask 'is it better', and the second question has no HTTP status code. Shadow traffic, offline eval gates and quality canaries replace the 500-rate dashboards you cannot use.

Both workloads are legitimate and they want opposite things from the same GPU. Combine tenant rate limits with fair admission and measured capacity reservations; separate fleets are an option when shared scheduling cannot meet the targets.

Asking nicely in the prompt gets you most of the way and fails on the tail, which is where the automation breaks. Constrained decoding can guarantee the shape of completed output for supported schemas. Interrupted generations still need a failure path, and compilation has a latency cost.

Scale to zero is correct for some workloads and an outage generator for others, and the deciding number is what the first request after idle experiences. Cold start turns an availability promise into a queue.