scheduling

Every ops engineer schedules jobs; few can say which shell cron uses, where stdout lands, or why percent signs delete half a command. Those gaps are exactly what interviews test.

The pair everyone configures and few can explain. QoS hints at eviction risk, but usage versus requests and Pod priority determine node-pressure ranking, and CPU and memory behave completely differently when you exceed them.

The friendliest DaemonSet question there is, and it still separates people who have operated one from people who have read about one. The scheduling rules hide the traps.

Multiple schedulers are supported, rarely understood, and dangerous in a specific way. The strong answer names the deadlock that gang scheduling fixes and the capacity race it can open.

Priority is one number with three separate effects: scheduling order, preemption victim selection, and eviction order under node pressure. Conflating those is the trap; designing tiers that survive both a full cluster and a hot node is the pass.

Anti-affinity is binary and falls over at scale; topologySpreadConstraints give you numeric skew control across labeled domains. The subtle parts are eligibility filtering before the skew math, and the fact that Kubernetes never rebalances anything for you.

Hard versus soft rules, topology keys, weights that lose to everything else, and the IgnoredDuringExecution half nobody remembers. Placement shapes blast radius only when paired with disruption budgets.

Driver, container toolkit, device plugin, DCGM and discovery must version-match on every node. The operator reconciles that ladder from one ClusterPolicy CR; MIG, upgrades and host kernels are where it bites.

Kubernetes does not see GPUs, it sees an extended resource that something has to advertise. When that advertisement is missing or wrong, the scheduler is behaving correctly and the node is lying.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

Cron is the most-deployed scheduler on earth and the worst-behaved at scale. The scored content is everything crontab cannot do: exactly-once firing, thundering herds, and what happens when the scheduler itself dies.

The question that separates people running GPU clusters from people running Kubernetes clusters that happen to have GPUs. The answer is that an integer count cannot express what a GPU workload needs.

Hard quotas strand idle GPUs and soft quotas mean the loudest team owns the cluster. Borrowing with reclaim is the shape that works, and the interesting decisions are all about what happens when the lender comes back.

A mixed fleet is the normal state, not a transitional one, because capacity arrives when it arrives. The failure is a scheduler that treats two very different cards as the same countable unit.

Prefill and decode are two different workloads sharing one GPU. Their resource demands differ, and an unchunked long prefill can delay active decodes. Profile the actual model and batch shape before choosing a scheduling change.