29We report one MTTR number and it never improves. Break down where incident time actually goes.▼medium★ EssentialNewUberNetflixPhonePe◆ premiumMTTR is three clocks hiding in one number. Profile detection, diagnosis and fix separately from incident timestamps; attack whichever clock dominates, which is almost never rollback speed.Open full answer →
32How often should a large training run checkpoint, and what does the answer depend on?▼hardNewNVIDIAMetaMicrosoft2 replies◆ premiumCheckpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.Open full answer →
