06Your batch fleet runs on spot and keeps losing nodes mid-job. How do you make interruptions survivable?▼mediumNewAmazon & AWSFlipkartDatadogunlockedEveryone quotes the discount. Fewer candidates can describe what happens when a notice arrives late or a worker disappears without completing shutdown and why some workloads shrug it off while others lose hours of compute.Open full answer →
30Your distributed training run died six hours into an epoch on preemptible GPUs. What happens next, and how do you design for it?▼mediumNewNVIDIAMetaDatabricks◆ premiumA training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.Open full answer →
34At 512 GPUs something fails every few hours. How does a training run survive that?▼expertNewNVIDIAMetaMicrosoft2 replies◆ premiumFailure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.Open full answer →
