CrashLoopBackOff investigation: find why the process exits
Investigate restarting containers using previous logs, termination reasons and probe events. Distinguish application exits, memory kills and startup failures.
TL;DR: CrashLoopBackOff describes delayed restart attempts after repeated container exits. Preserve the previous attempt's evidence and determine what ended it before changing probes, memory limits or restart settings.
Treat the displayed status as a symptom
A container starts, exits and is restarted according to the applicable policy. Repeated failures can trigger a backoff before the next attempt. CrashLoopBackOff is a useful display reason, not a complete root cause or a Pod phase. Kubernetes explains container states and restart behavior in its Pod lifecycle documentation.
First identify the container. A Pod can have an application container, init containers and sidecars, and the failing component may not be the one producing the logs you first opened. Record the namespace, Pod UID, container name and image digest so evidence from a replacement Pod is not mistaken for the original failure.
Collect the last attempt before replacing anything
For an authorized workload named billing-worker-abc in namespace billing, these read-only commands inspect the current state and previous container attempt. Replace the example names with the actual workload.
kubectl -n billing describe pod billing-worker-abc
kubectl -n billing logs billing-worker-abc -c worker --previous --timestamps
kubectl -n billing get pod billing-worker-abc -o yaml
The Kubernetes debugging guide describes Pod inspection and log collection. Previous logs are especially useful when the current attempt has only just started. Availability depends on retained container logs; use centralized logging if the node no longer has the relevant attempt.
Protect collected manifests and logs because they can reveal configuration or customer data. Inspect secret references without printing secret values into incident chat.
The decision graph prioritizes evidence. More than one cause can exist, so retain the timeline when failures change after a mitigation.
Match the termination to a hypothesis
| Evidence | Hypothesis to test | Mistake to avoid |
|---|---|---|
| Application reports missing configuration and exits | Deployment omitted or renamed an input | Increasing memory without evidence |
| Last termination reason is OOMKilled | Memory use exceeded an enforced boundary | Assuming a larger limit fixes a leak |
| Probe failures precede termination | Health policy ends an otherwise running process | Disabling all health checks permanently |
| Exit code zero with repeated restarts | Long-running workload actually finishes its command | Treating every loop as a crash |
| Exit code 137 without clear cause | Process received SIGKILL or equivalent exit reporting | Declaring OOM solely from the number |
Exit code 137 often corresponds to termination by signal 9 in common reporting conventions. It is not sufficient evidence of an out-of-memory kill. Check the termination reason and available node or runtime evidence. Likewise, an application's final log line may be the last thing it wrote before an external kill, rather than the cause of that kill.
Work through a startup-probe failure
Consider a hypothetical release that loads a larger ruleset before serving requests. Startup now takes about 50 seconds in the test environment. A liveness check begins earlier and repeatedly fails while initialization is still progressing, leading to restarts before the process can become ready.
Confirm the sequence using process logs and probe events. Then use an appropriate startup probe and a startup budget supported by measurements. Keep readiness separate so traffic waits for useful service. Probe semantics explains why readiness failure does not itself request a restart.
Do not merely set every threshold to a large value. If startup is hanging on an unavailable dependency, a long allowance postpones detection without making progress. Record a meaningful initialization milestone and compare successful versus failing attempts.
For a configuration regression affecting many replicas, restoring a known compatible configuration or image may be the quickest mitigation. Verify rollback compatibility when the release also changed schemas or external state.
Prove the service recovered, not just the process
After the change, observe restart count over a relevant interval and exercise the actual operation. A container can stop restarting because its entrypoint now sleeps forever while the application remains unavailable. Check readiness, request outcomes and the original failure trigger.
In an interview, explain which observation eliminated each competing hypothesis. “I checked logs” is weaker than “the previous attempt completed initialization, then a liveness failure preceded SIGTERM, so I investigated the probe rather than the image pull.” Use diagnostic method to keep those claims testable.
Self-check: the Pod is Running and restart count has stopped increasing, but readiness remains false. Is the incident resolved?
No. Running describes lifecycle state, and restart stability describes process survival. Determine why the application is not ready and verify useful traffic before declaring recovery. Preserve the distinction in the incident update so users do not hear that service is restored merely because the restart loop stopped.