07Training throughput is 40 percent below what the GPU count predicts. The GPUs look fine. Where do you look?▼expertNewNVIDIAMetaMicrosoft2 repliesunlockedAt sixteen GPUs and above, the network is the computer. Collectives run at the speed of the worst link in the group, and placement decides which link that is.Open full answer →
34At 512 GPUs something fails every few hours. How does a training run survive that?▼expertNewNVIDIAMetaMicrosoft2 replies◆ premiumFailure stops being an event and becomes a rate. Once mean time between failures drops below the length of a run, the design question is not how to avoid interruption but how cheaply you can absorb one.Open full answer →
