StatefulSets and stable identity: ordinals, storage and safe replacement
Explain StatefulSet ordinals, headless Services and persistent claims. Reason about replacement, storage retention and database membership without assuming Kubernetes provides replication.
TL;DR: A StatefulSet gives each replica a stable ordinal identity and can associate it with persistent storage. The application still owns replication, quorum and the rules that prevent two processes from acting as the same writer.
Identity survives replacement; the process does not
A three-replica StatefulSet named ledger normally creates ledger-0, ledger-1 and ledger-2. Replacing ledger-1 preserves its logical ordinal, but creates a new Pod object with a new UID. Its IP can change. Applications should not treat a previously observed Pod IP as a permanent member address.
A headless Service provides the governing network identity used for per-Pod DNS. Persistent volume claim templates can allocate a claim for each ordinal. Kubernetes's StatefulSet documentation describes these identities and the default ordered startup behavior. DNS caching can delay discovery after an object is created or replaced, so membership logic must tolerate transient absence.
This diagram shows ownership relationships, not data replication between claims. Two replicas with two volumes are not automatically two copies of the same database. The software running inside the Pods must establish replication and decide which member can accept writes.
Ordered startup can reveal a readiness deadlock
With the default OrderedReady policy, later replicas wait for earlier replicas to become ready. That is useful when initialization genuinely requires a predecessor. It can also deadlock an application whose readiness probe insists that every peer already exists before any member becomes ready.
Imagine ledger-0 refuses readiness until it discovers all three members. The controller waits for ledger-0 before creating ledger-1. Neither side can progress. Review the application's bootstrap protocol and readiness definition before changing the pod management policy. Parallel creation can be appropriate for a peer-discovery protocol, but it does not repair an application that cannot tolerate concurrent membership changes.
| Identity or mechanism | Kubernetes supplies | Application responsibility |
|---|---|---|
| Ordinal name | Stable logical replica label | Associate it with safe member state |
| Per-replica claim | Persistent storage association | Consistent data and recovery format |
| Ordered readiness | Controller sequencing | A bootstrap condition that can succeed |
| Replacement Pod | Recreated workload object | Membership, fencing and catch-up |
A retained claim is a recovery decision
By default, StatefulSet storage is retained through common scale-down or deletion operations. Supported releases also provide a PVC retention policy, so inspect the actual object rather than relying on a remembered default. StorageClass reclaim behavior and the underlying volume's lifecycle remain separate controls. The persistent volume documentation explains that distinction.
Suppose the workload scales from three replicas to two and later returns to three. Reusing the old ordinal's data can accelerate recovery, but the member may be far behind or belong to an obsolete cluster identity. The application must decide whether to catch up, reinitialize or reject that data. Deleting the claim to “fix the Pod” can destroy the only recoverable copy if replication was already degraded.
Record the claim-to-volume mapping, storage topology and recovery procedure before a destructive repair. A volume restricted to one zone can keep a replacement pending in another zone even when the node has enough CPU. This is a placement constraint, not evidence that the StatefulSet controller forgot to restart the Pod.
For planned node maintenance, PodDisruptionBudgets and drains explains how unhealthy members and unschedulable replacements reduce the number of safe evictions.
Treat force deletion as a membership event
If a node is unreachable, the control plane may be unable to prove that its process stopped. Forcing deletion and starting a replacement can violate an application's assumption that only one instance holds a particular identity. ReadWriteOnce storage access is not a universal database fencing protocol; its behavior and scope do not substitute for rejecting stale writers.
Use the application's documented failure and fencing procedure. Check whether the old instance can still reach the database or shared storage, and how authority transfers. The fencing example follows a suspended writer that resumes after ownership has moved to another client. Storage must reject its obsolete authority.
Self-check: all three StatefulSet Pods are ready and each has a bound claim. Can you promise survival of one data-volume loss? No. You need evidence of application replication, quorum behavior and a tested recovery path. Kubernetes has established the declared workload and storage associations; the database's durability guarantee requires additional mechanisms and verification.