04One node in your training fleet makes every job it touches 30 percent slower, but it passes health checks. How do you find and handle it?▼hardNewNVIDIAMetaMicrosoft2 repliesunlockedGPUs fail gradually before they fail loudly. Xid errors, ECC retirement, thermal throttling and a degraded NVLink all produce a node that is up, schedulable, and slowing every gang it joins.Open full answer →
30Your distributed training run died six hours into an epoch on preemptible GPUs. What happens next, and how do you design for it?▼mediumNewNVIDIAMetaDatabricks◆ premiumA training job is an hours-long computation with no user watching. Whether an interruption costs four minutes or four days is decided entirely by checkpoint cadence and restart semantics designed before the run.Open full answer →
33A training run from three months ago cannot be reproduced. Same code, same data, different result. Where does the difference come from?▼hardNewNVIDIADatabricksGoogle2 replies◆ premiumCode and data are the two things people pin, and they are rarely the two things that changed. The driver, the framework build, the kernel selection and the data ordering all vary underneath a pinned repository.Open full answer →
35Your GPUs sit at 40 percent utilisation during training and the model code is fine. What is starving them?▼hardNewNVIDIAMetaDatabricks2 replies◆ premiumExpensive accelerators waiting on cheap storage is the most common form of waste in a training cluster, and it looks exactly like a code problem on every dashboard.Open full answer →
