hardNewNVIDIAMetaMicrosoft

How often should a large training run checkpoint, and what does the answer depend on?

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

Checkpoint too rarely and a failure costs hours of GPU time. Checkpoint too often and you pay for stalled GPUs writing terabytes to storage. There is an arithmetic answer and most teams never do it.

20 answers per topic instead of 10, and your progress kept · no cardor unlock all 390 remaining answers · ₹2,000 / $25

UP NEXT ON YOUR JOURNEY

Next in this trackA training run from three months ago cannot be reproduced. Same code, same data, different result. Where does the difference come from?Next in this trackAt 512 GPUs something fails every few hours. How does a training run survive that?Next in this trackYour GPUs sit at 40 percent utilisation during training and the model code is fine. What is starving them?

DISCUSSION · 0

Nothing here yet. Say how you would answer it.