The cloud control plane is a dependency you forgot to design around

TL;DR: Separate what runs your traffic from what changes your infrastructure. A control-plane impairment can leave existing data-plane operations healthy. A regional outage can also damage both. Recovery that requires launching, scaling or reconfiguring must account for unavailable APIs. Pre-provision the capacity you would need, and prefer failovers that require no API call at all.

Two planes, two reliability profiles

The data plane is what carries your traffic: the instances already running, the load balancers already configured, the DNS already resolving. Its failure modes differ from those of management APIs; consult the specific service and operation.

The control plane is what changes things: launching an instance, modifying a security group, updating a record, promoting a replica. It is a distributed system with its own dependencies, and it can be impaired independently. Recovery demand can also encounter throttling or capacity shortages.

Providers document this distinction and most architecture diagrams ignore it.

What that means for a recovery plan

Read any runbook and count the steps that require an API call to succeed. "Scale up the standby fleet." "Update the DNS weight." "Promote the read replica." "Fail over to the other region." Each one is a control-plane operation, and each is being attempted by every other customer in that region at the same moment.

The result is the failure people describe as the cloud provider being down when their instances were fine. The instances were fine. The plan needed an API that was not.

Static stability

The principle is that the system holds its current state, and continues to work, without needing to make a change. Concretely:

• Pre-provision the failover capacity. If losing a zone means the remaining two must serve everything, run them at a level where they already can. Capacity that has to be launched during the event is capacity you may not get, both because the API is degraded and because everyone else wants the same instance type.
• Prefer health-check failover over reconfiguration. A load balancer that stops sending traffic to unhealthy targets needs no API call. A script that removes a target group does.
• Cache what you depend on. A service that fetches configuration or credentials from a provider API on every start will not start during the event. Cache configuration only where stale values are safe. Do not extend expired credentials or bypass revocation; plan credential refresh and the permitted failure behavior separately.
• Know which of your failovers are data-plane operations. Anycast and health-checked DNS behave differently from a record update. One is already running; the other is a change.

The uncomfortable arithmetic

Static stability costs money, and that is the whole trade. With three equally sized zones, surviving one lost zone at the design load requires total capacity of at least 1.5 times that load: 50 percent extra relative to demand, or one third of provisioned capacity spare. Allow more for uneven routing and latency limits. Autoscaler headroom is a workload-specific policy, not a fixed 10 percent.

The counter-argument is that a design that provisions during recovery accepts API, capacity and startup dependencies at that moment. Making that trade explicit, with the cost of both sides on the slide, is what a senior answer looks like.

The test

Ask of any resilience plan: which steps require a control-plane call, and what happens to each if that call fails or takes twenty minutes? A plan with no such steps is statically stable. A plan with three of them has three ways to not work on the day, and the honest version of the plan says so.

Self-check

Three zones each provide 100 units of usable capacity. What load survives loss of one without scaling? At most 200 units before additional safety margin. That is two thirds of the original 300, not 50 percent utilization.

Sources: AWS static stability.