Circuit breakers and bulkheads: contain dependency failures
Separate failure detection from capacity isolation. Design breaker recovery probes and bounded dependency pools with a worked shared-worker example.
TL;DR: A circuit breaker temporarily stops calls to a failing dependency. A bulkhead limits how much capacity that dependency can occupy. Use explicit recovery probes and bounded queues so failure handling does not consume the resources needed for healthy work.
A timeout alone can leave every worker occupied
Imagine an API with 100 request workers. Most requests call a recommendation service before returning. That dependency slows to the full timeout, and new requests continue arriving. Even if each call eventually times out, the API can spend the entire interval holding workers for calls that are unlikely to succeed.
A breaker observes selected outcomes and changes whether new calls are attempted. A bulkhead reserves or caps concurrent work by dependency, tenant or operation. Microsoft's circuit-breaker pattern and bulkhead pattern explain these distinct controls.
The two mechanisms address different resources. A breaker cannot reclaim a worker already blocked in an uncancellable call. A bulkhead cannot decide that a dependency has recovered; it only limits how much work is admitted.
Make breaker transitions measurable
In the closed state, calls are permitted and their outcomes contribute to the policy. When the configured failure condition is met, the breaker opens and rejects new attempts for a period. A half-open state permits a limited recovery sample before returning to normal traffic or reopening.
Define what contributes to failure. An invalid customer request should not usually trip a shared dependency breaker. A timeout or eligible server error may. Record a minimum sample count and observation window so one failure in a tiny traffic sample does not accidentally look like a broad outage.
For a hypothetical policy, the team might require at least 20 eligible calls and more than half failing in its window. Those numbers are exercise inputs, not recommended defaults. Evaluate detection speed against the dependency's capacity and the caller's tolerance for rejected requests.
Half-open concurrency must be bounded. If every process releases a large probe batch at the same instant, the recovery test becomes a new load spike. Consider the number of caller replicas, stagger recovery where appropriate and coordinate with the dependency's advertised recovery limits.
Protect the shared worker pool
Return to the 100-worker API. Cap recommendation calls at 20 concurrent executions and allow at most ten additional requests to wait for that capacity. Once both bounds are reached, use the operation's defined fallback or reject the request promptly.
This bounds the resources allocated to that path only if the implementation actually releases or avoids occupying shared workers while waiting. An asynchronous semaphore does not guarantee safety if an unbounded upstream queue retains large request bodies or ties up another constrained pool.
| Control | Capacity it should bound | Evidence to observe |
|---|---|---|
| Dependency concurrency limit | Calls executing against one dependency | Active count and completion latency |
| Waiting-queue limit | Work awaiting a slot | Queue length and oldest wait |
| Request deadline | Total useful request lifetime | Cancellation and deadline exhaustion |
| Breaker | New attempts during likely dependency failure | State transitions and rejected calls |
| Retry budget | Additional attempts beyond original demand | Attempt-to-request ratio |
A pool of 20 recommendations plus another pool of 90 reporting calls can still exceed a shared limit of 100. Review the sum and the scheduling model. Partitioning requires an explicit budget, not merely a semaphore at every call site.
Choose a fallback that preserves the operation's meaning
For product recommendations, returning a page without suggestions may be acceptable. For payment authorization, returning success without authorization is not a valid fallback. Define the degraded result with the product owner and make it visible in telemetry so “successful” fallback responses do not hide prolonged dependency failure.
A cache fallback also needs a freshness contract. Serving an old product description may be tolerable; serving an old account balance as current can mislead the user. Include provenance or age where the consumer needs it to interpret the answer.
Order retries and breakers deliberately. If a retry wrapper bypasses the breaker for later attempts, an open state may not suppress the load you expected. If every local rejection is counted as a fresh remote failure, the measurements also become misleading. Trace one request through the actual library composition and test the emitted counters.
Exercise recovery under concurrent demand
During a controlled test, hold the dependency slow, keep healthy routes busy and verify that their latency remains acceptable. Then restore the dependency gradually. Observe whether probes succeed, queued work expires correctly and retry traffic stays within its allowance. Retry backoff and jitter gives the attempt-level calculation.
Self-check: the breaker opens, but the API still has no free workers for thirty seconds. Does that prove the breaker is broken?
No. Calls admitted before the transition may still occupy workers until their deadlines or cancellation complete. Inspect in-flight work and the queue, then verify the bulkhead and cancellation behavior. Opening a breaker is an admission decision; recovery also depends on releasing already consumed capacity.