Kubernetes probes and readiness gates: startup, restarts and traffic
Choose Kubernetes startup, liveness and readiness checks by the action they trigger. Avoid restart storms during dependency failures and explain custom readiness gates.
TL;DR: Startup checks protect slow initialization, liveness checks decide when a container needs restarting, and readiness checks decide whether a Pod should receive ordinary Service traffic. Choose each check according to the corrective action it triggers.
Different failures require different actions
A stuck process may recover after restart. A healthy process waiting for a failed shared database usually will not. If liveness depends on that database, every replica can restart together and add connection and initialization load while the database is already struggling.
A configured startup probe delays liveness and readiness probing until startup succeeds. Readiness failure marks the Pod unready rather than restarting the container. Liveness or startup failure can lead to restart under the relevant container restart behavior. Kubernetes documents these distinctions in its probe reference.
Probe timing should reflect measured application behavior. A very short timeout can misclassify a loaded but recovering process; an excessively long failure window can leave an unrecoverable instance consuming capacity. State the workload and failure being detected before choosing numbers.
| Check | Question | Typical action after sustained failure |
|---|---|---|
| Startup | Has initialization completed? | Restart after configured failure policy |
| Liveness | Is this container in a state restart can repair? | Restart the container |
| Readiness | Can this Pod serve its intended traffic now? | Exclude it from ordinary ready Service endpoints |
| Custom readiness gate | Has an external condition been confirmed? | Keep Pod readiness false until conditions satisfy the gate |
Dependency failures need a decision
The last decision is application-specific. A catalog service may serve cached reads while its refresh dependency is down. A payment writer unable to persist an operation should not claim that it can accept that operation successfully.
Worked example: every Pod becomes unready
Consider a search API with ten replicas and a shared indexing dependency. The API can answer existing queries from a local snapshot while new index refreshes fail. A readiness endpoint that requires a fresh indexing connection removes all ten replicas at once, turning stale-but-useful search into complete unavailability.
Define the promised behavior. If the product accepts a snapshot up to a stated age, readiness can depend on having a valid usable snapshot rather than the refresh service's instantaneous availability. Monitor freshness separately and make the degraded state visible. Once the snapshot crosses the accepted limit, the readiness decision changes.
This does not mean dependencies should never influence readiness. If a replica cannot serve any valid request without a dependency, remaining ready may only send users guaranteed failures. The important point is to model the actual serving contract and system-wide consequence.
Load-test the health endpoint too. An endpoint that opens an expensive new database connection on every probe can become a source of load, especially across many replicas. Keep checks bounded and avoid generating work that is disproportionate to the information needed.
Readiness includes more than the probe result
Pod readiness can incorporate custom conditions through readiness gates, for example when an external controller must confirm that traffic registration is complete. The responsible controller must update the condition correctly; a missing required condition can leave a Pod unready despite a passing container readiness probe.
The Pod lifecycle documentation explains readiness gates and conditions. Inspect the full Pod status, rather than equating a healthy HTTP endpoint with the complete readiness decision.
Readiness also does not guarantee immediate termination of every existing connection. Endpoint updates propagate through networking components, and existing sessions have their own lifetime. Combine readiness with appropriate termination and drain behavior. Services and endpoints explains the traffic-selection path.
If restarts have already begun, use the CrashLoopBackOff investigation to distinguish probe termination from an application exit or memory kill before changing thresholds.
Prove the chosen response
For a slow-starting application, measure a representative cold initialization and allow a bounded startup window. For a deadlock, verify that liveness detects the condition and restart actually restores service. For a shared dependency outage, observe whether the policy preserves useful capacity or creates a cascade.
Should liveness query every downstream service? Usually no. A dependency failure that restart cannot repair should not trigger synchronized restarts.
Can a Deployment have running Pods but no ready replicas? Yes. Running describes lifecycle state; readiness describes suitability for serving under the configured checks and conditions. Review Pods, ReplicaSets and Deployments when interpreting rollout status.
Does increasing initial delay solve every slow startup? It can hide a timing symptom. Prefer an explicit startup check with measured limits, then investigate initialization regressions separately.