volcano

Multiple schedulers are supported, rarely understood, and dangerous in a specific way. The strong answer names the deadlock that gang scheduling fixes and the capacity race it can open.

The default scheduler places pods one at a time, which is exactly wrong for a job that needs all sixteen or none. Two half-placed jobs can hold each other's GPUs forever without either ever starting.