The control loop: the one idea Kubernetes is built on

TL;DR: Kubernetes is a set of independent controllers, each watching the API server and reconciling one difference between desired and actual state. Independent reconciliation lets one controller resume after another recovers. Failures can appear as stalled progress, status conditions, events or component errors.

The shape

A controller does exactly this, forever:

watch(resource)              # observe desired state
observe(reality)             # observe actual state
if desired != actual:
    act to close the gap

The deployment controller watches Deployments and creates ReplicaSets. The replicaset controller watches ReplicaSets and creates or deletes Pods. The scheduler watches Pods with no nodeName and writes one. The kubelet watches Pods bound to its node and makes containers exist. The EndpointSlice controller watches Pods and Services and updates EndpointSlices.

These controllers coordinate desired state through API objects. Their actions also use other interfaces: kubelet calls the container runtime, and storage controllers call storage APIs. Etcd stores Kubernetes API state; application data, container images and node-local state live elsewhere.

Why the architecture was chosen

Three consequences fall out of it, and all three are the actual reason the model is used.

Level-triggered rather than edge-triggered. Events usually trigger reconciliation, which checks the current difference rather than blindly replaying a transition. If a controller is restarted, or misses an event entirely, it recomputes the difference on its next pass and still converges. That is why Kubernetes tolerates its own components being down.

The control plane is not on the data path. Kill the scheduler and running pods keep serving traffic, because the kubelet does not need the scheduler for anything already bound. New pods stay Pending. During an API-server outage, many existing workloads and programmed network paths keep working. Workloads that need live API reads, fresh credentials or new storage/network setup may fail; continued service is conditional on their dependencies.

Extensibility uses the same pattern. A custom resource plus a controller is architecturally identical to a built-in one. That is what an operator is: a control loop that encodes the operational knowledge for a specific piece of software, using the same API conventions. It still needs carefully scoped RBAC, retry handling and operational support.

What this predicts about failures

A broken reconciliation loop often leaves an object waiting for progress. Look for status conditions and events explaining that wait, then inspect the responsible component's logs or external dependency.

A Pending pod may be unscheduled or already assigned to a node while images, volumes or init containers are prepared. Check PodScheduled and spec.nodeName first. Terminating is a display status for deletion in progress, not a Pod phase: grace periods, unreachable nodes and finalizers can all delay deletion. A PVC may be protected while a pod still uses it; inspect references and finalizers before removing any protection. A Deployment reporting fewer ready replicas than desired means some loop downstream is blocked and the status is telling you where.

The diagnostic instinct that follows: when something is stuck, find the controller responsible for that transition and read its view of the world, which is almost always in the object's events and status conditions rather than in a log file.

Self-check

How do durable desired state and repeated reconciliation let scheduling resume after a scheduler restart? A remote-call-based system can also recover if it persists work and retries safely; the distinction is the recovery protocol, not whether remote calls exist. If a PersistentVolumeClaim is stuck Terminating, which mechanism is holding it, and what would you check before removing it by hand?