Kubernetes operators and CRDs: API contracts, reconciliation, and deletion
Explain how CRDs extend the Kubernetes API, how operators reconcile custom resources, and why finalizers and storage-version migrations require operational care.
TL;DR: A CustomResourceDefinition adds a resource type to the Kubernetes API. An operator supplies the controller logic that reacts to instances of that type. Installing the definition can make an object valid and visible to
kubectlwhile leaving its requested database or backup completely unimplemented because the controller is absent.
That separation is the first thing to inspect when a custom resource exists but nothing happens. The API server accepts and stores the request. A controller must then observe it, obtain the permissions it needs, and reconcile the external or Kubernetes resources that make the request true. Custom resources describes the API extension; the operator pattern describes the operational loop.
Follow one object through its owners
Consider an illustrative CacheFleet API. Its definition declares a namespaced resource, a structural schema, and the versions clients may use. A resource instance might look like this:
apiVersion: platform.example.com/v1alpha1
kind: CacheFleet
metadata:
name: catalogue-cache
namespace: workshop
spec:
replicas: 3
memoryMiB: 512
This is a custom-resource example, not a built-in Kubernetes object. It requires a matching CRD and controller. The schema should reject negative replica counts and invalid memory values; the controller must still handle valid requests that cannot currently be fulfilled, such as insufficient cluster capacity.
| Component | Responsibility | Failure evidence |
|---|---|---|
| CRD schema | Define accepted structure and field types | API validation rejection |
| Controller | Move actual state toward the requested state | Reconciliation errors, retry backlog |
| RBAC | Permit required API operations | Forbidden responses in controller logs |
| Status contract | Report progress and observed request version | Stale generation or an explanatory condition |
A useful status contract includes conditions and an observedGeneration value chosen by the API author. Compare it with metadata.generation: a Ready condition calculated for generation 4 does not confirm that a generation-5 resize succeeded. These fields need controller implementation and a documented meaning. Merely naming a status field does not make the API server maintain it.
Reconciliation has to survive repetition
Imagine that the controller creates a cloud cache, then crashes before recording its identifier. On restart, a naive controller creates another cache. A stable external identity or provider-supported idempotency key lets the next reconciliation discover the existing resource and continue. The same issue occurs when the API request times out after the provider has accepted it.
Keep status updates separate from changes that trigger expensive external work. Bound retries, classify permanent validation failures, and expose enough detail to identify the object that is failing. Avoid logging credentials embedded in provider responses. For owned Kubernetes objects, owner references can support garbage collection, but external resources need an explicit lifecycle policy.
The Kubernetes control loop explains the underlying desired-state model. An operator adds domain knowledge, such as how to resize safely or verify a backup, rather than changing that model into a once-only script.
Deletion and version changes are production operations
A finalizer lets a controller finish cleanup before the API object disappears. Deletion sets a timestamp; the responsible controller performs its cleanup and removes its finalizer. If the controller is down or its cloud permission has expired, deletion may remain pending. Read the finalizer lifecycle before treating a stuck object as disposable metadata.
Removing the finalizer manually can orphan the external cache and its bill. First identify the cleanup obligation, inspect the external resource, and repair the controller or perform the documented recovery. A stateful service may deliberately retain data; deletion policy must distinguish deleting the control object from deleting that data.
Version changes need similar care. A CRD can serve multiple versions while using one storage version. Changing which version is marked for storage does not rewrite all existing objects. Conversion webhooks also become an availability dependency when requests need conversion. Follow a migration that rewrites stored objects and verifies old-version retirement rather than deleting the old schema immediately. CRD versioning covers these steps.
A diagnostic exercise
A new CacheFleet has existed for ten minutes. Its generation is 7 and status says Ready for generation 6. Where do you look? Check controller health, its watch scope, RBAC, and the reconciliation errors for this object. The stored request is newer than the controller's reported result. Reapplying identical YAML offers little evidence.
A deleting object has a finalizer, and the external cache still exists. Can removing the finalizer fix the cache leak? It removes the API's cleanup barrier. It does not perform cleanup. Restore or carry out the cleanup operation, verify the intended retention outcome, then complete deletion.