10Design a GPU serving platform for several LLMs with autoscaling and a cost ceiling.▼hard★ EssentialNewNVIDIADatabricksUber2 repliesunlockedThe fastest-growing design round in infrastructure hiring. GPU economics break the assumptions CPU autoscaling is built on, and the answer has to start from that rather than from Kubernetes.Open full answer →
11A vLLM pod serving an LLM OOMs at a request rate it handled fine yesterday, dropping in-flight requests. Walk me through it.▼hard★ EssentialNewDatabricksNVIDIAGoogle○ sign inCPU memory intuition fails on inference servers. The scarce resource is KV-cache, requests arrive with unseen context lengths, and the fix is queue-depth scaling plus admission control, not bigger pods.Open full answer →
14You have eight 80GB GPUs and a 70B model. Tensor parallel across all eight, or two replicas of four? Defend the choice.▼expertNewNVIDIADatabricksMeta2 replies○ sign inBoth configurations fit the weights. Compare cache capacity, collective overhead and replica placement under the actual traffic shape; neither layout wins every workload.Open full answer →
