← 🤖 AI Infrastructure
Core
Model-serving batching and queueing: throughput within latency bounds
Tune inference batching with first-token and inter-token latency in view. Work through token budgets, queue limits, cancellation and representative load tests.
a free account opens the core tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
AI & GPU InfrastructureA vLLM pod serving an LLM OOMs at a request rate it handled fine yesterday, dropping in-flight requests. Walk me through it.→AI & GPU InfrastructureYou have eight 80GB GPUs and a 70B model. Tensor parallel across all eight, or two replicas of four? Defend the choice.→AI & GPU InfrastructureLong prompts are making short requests slow on the same replica. What is happening, and how would you fix it?→AI & GPU InfrastructureSpeculative decoding promises a big latency win. When does it not deliver, and what does it cost you?→Cloud Platforms & ArchitectureYour service autoscales on CPU and still falls over during traffic spikes. Why, and what would you scale on instead?→DevOps System Design & ArchitectureAn e-commerce platform takes 10x traffic during flash sales and the database locks up. Fix it.→