Kubernetes QoS and eviction: distinguish node pressure from container OOM
Learn Kubernetes resource QoS, node-pressure eviction and container OOM boundaries through competing pod examples and a practical evidence checklist.
TL;DR: QoS describes resource declarations. Diagnose eviction from the pressure signal, requests, actual usage and priority; distinguish it from a process killed at its memory boundary or CPU work being throttled.
Three failures can look like one restart
A container exceeds the memory its cgroup can support, reclaim fails, and the kernel kills a process. Separately, a node can run short of memory or disk and the kubelet can evict a whole pod. CPU quota exhaustion causes throttling rather than an OOM kill. These mechanisms have different evidence and different fixes.
A Deployment may replace an evicted pod, while a container restart can happen inside the same pod. Record the pod UID, container's previous termination reason and node events before restarting everything. Losing the original evidence makes a successful replacement look like a diagnosis.
A reason such as OOMKilled narrows the investigation but does not independently establish whether the trigger was container-local or node-wide. Correlate runtime status with kernel and node evidence. Start from Linux memory accounting when the reported usage looks contradictory.
Classify the admitted pod
For conventional container-level CPU and memory declarations, the Kubernetes QoS rules yield these classes:
| Class | Resource declarations | What it does not promise |
|---|---|---|
| Guaranteed | Every container has positive CPU and memory requests equal to its limits | Survival through every node failure or pressure condition |
| BestEffort | No container has CPU or memory requests or limits | Access to spare resources when others need them |
| Burstable | Other combinations | A fixed place in an eviction queue |
Inspect the stored pod because admission defaults can change submitted resource fields. The table deliberately uses container-level declarations; pod-level resource support and its QoS interactions depend on the cluster version and configuration. A sidecar with different declarations can change the pod's classification. Copying the application's resource block alone is insufficient.
Requests already affect scheduler placement. During node pressure, they also provide a reference for judging usage. This is why lowering requests to fit more pods can change which workload absorbs a memory shortage.
Predict an eviction without memorizing a class order
The kubelet first attempts applicable node-level reclamation. If pressure persists, its documented pod ranking considers whether usage exceeds requests, then priority, then usage relative to requests. QoS is a useful clue for memory pressure, but the kubelet does not simply sort on the class label. Disk, inode and PID pressure have different accounting constraints. See node-pressure eviction.
Consider three fictional pods on a memory-pressured node. Assume all are ordinary evictable pods, node-level reclaim has failed and priority numbers are equal.
| Pod | Memory request | Observed memory | Interpretation |
|---|---|---|---|
| Ingest | 512 MiB | 1,536 MiB | Exceeds its request by 1,024 MiB |
| Search | 2,048 MiB | 1,024 MiB | Uses less than requested |
| Metrics | 256 MiB | 512 MiB | Exceeds its request by 256 MiB |
Ingest and Metrics enter the over-request group before Search. Raising Ingest's priority can change its ordering within that group, but does not make memory appear. The exact choice also depends on the pressure signal and current measurements. A recommendation to protect everything with maximum priority removes the useful distinction and leaves the shortage unresolved.
Node-pressure eviction also differs from the voluntary eviction API: a PodDisruptionBudget does not prevent the kubelet from recovering a node under pressure. Preserve availability through spare placement capacity and replica distribution; a budget cannot reserve physical memory.
Verify the proposed repair
If Ingest has a repeatable 1.5 GiB working set, raising its request can make scheduling more honest. Before doing so, calculate whether its replicas and deployment surge still fit. Otherwise the repair replaces runtime eviction with Pending pods. If memory keeps growing for identical completed work, investigate retention before reserving an ever-larger amount.
For disk pressure, identify whether image storage, logs, writable layers or another filesystem consumer crossed the relevant threshold. CPU and memory QoS labels do not explain missing inodes. For a local OOM, compare the measured peak and allocation pattern with the cgroup limit; increasing node count alone may leave that limit unchanged.
An interviewer can test the boundary with this self-check: a Guaranteed pod stays under its memory limit, yet is evicted while the node reports disk pressure. Is that impossible? No. Guaranteed describes CPU and memory declarations and does not exempt the pod from disk-pressure eviction. The next evidence is filesystem capacity/inodes, ephemeral-storage accounting and the kubelet's eviction message, followed by whether a replacement has a healthy node to use.