03A multi-node training job sits Pending forever while the cluster shows free GPUs. What is happening?▼hardNewNVIDIAMetaDatabricks2 repliesunlockedThe default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.Open full answer →
06Six teams share a GPU cluster. Two of them are idle most nights. Design the quota model.▼hardNewDatabricksUberMicrosoft2 repliesunlockedHard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.Open full answer →