Load balancer health and draining: stop admission before shutdown
Distinguish failed health checks from deliberate deregistration. Plan request draining, shutdown budgets and ALB fail-open behavior with a worked rollout timeline.
TL;DR: Health checks decide which targets appear eligible; draining coordinates the removal of a target while work is in flight. Align new-request admission, load balancer behavior and application shutdown so the process remains available for the work you intend to finish.
Health is a sampled observation
A load balancer probes a configured endpoint and evaluates responses against its health policy. That endpoint must represent the target's ability to serve the intended traffic. A handler that always returns success can stay green while the application rejects every useful request. A handler that fails for every minor dependency issue can remove too much capacity at once.
AWS documents probe configuration and target states in ALB target health checks. One operationally significant detail: when all registered targets in an ALB target group are unhealthy across the enabled Availability Zones, ALB can fail open and route to them. Do not treat unhealthy status as a universal emergency traffic kill switch.
Probe intervals and consecutive-failure thresholds also create detection delay. Exact timing depends on the phase of the next probe, response time and the implementation. “Three failures at ten-second intervals means exactly thirty seconds” is an unjustified precision claim.
Deliberate removal has a different objective
Deregistration tells the load balancer that a target is leaving. A draining period gives eligible in-flight work time to finish while new routing moves away. The ALB target-group attributes guide explains deregistration delay and related behavior.
The sequence is conceptual. A real orchestrator may initiate endpoint removal and process termination concurrently, so the application must tolerate propagation time. Confirm the platform's actual sequence instead of relying on diagram order as a timing guarantee.
Budget a bounded drain
Consider an illustrative API whose supported request duration is at most 20 seconds. The team measures a possible five-second routing propagation interval and wants another five seconds for cleanup. A shutdown budget must cover at least those assumed components, with an appropriate margin based on measurement. A 15-second process grace period cannot preserve a request that may legitimately run for 20 seconds after the transition begins.
The load balancer's wait and the process's grace period must agree. Setting a long deregistration delay while terminating the process immediately does not preserve requests. Conversely, keeping a process alive after the load balancer closes its path may consume capacity without helping clients.
| Work type | Drain decision | Evidence to test |
|---|---|---|
| Short HTTP request | Finish within a bounded grace period | Response completes without reset |
| Long download | Allow a documented limit or support restart/resume | Client recovery behavior |
| WebSocket or streaming session | Signal reconnect and bound lifetime | Reconnect load and session continuity |
| Background job triggered by request | Track durable job ownership separately | No lost acknowledgment or duplicate effect |
A client can retry after a broken connection even if the server completed a write. Preserve idempotency where repeating the operation would otherwise duplicate an effect.
Preserve capacity while removing targets
Suppose four equal-capacity targets are operating near their safe concurrency limit. Removing two simultaneously may overload the survivors before replacement targets are ready. The drain procedure can be correct for individual requests and still cause a fleet-wide failure through reduced capacity.
Choose rollout concurrency based on the surviving capacity and request distribution. Confirm new targets are genuinely ready before depending on them. Health checks should exercise the configured port and route, while separate application signals establish whether useful traffic succeeds.
For Kubernetes, connect target removal to readiness behavior and container termination. The load balancer, ingress controller, Service endpoints and process can each observe a change at different times. Trace those observations in a controlled rollout.
Test the worst eligible request
During a staging exercise, start a request near the supported duration limit and remove its target. Verify when new requests stop arriving, whether the existing request completes and how the process exits. Repeat for streaming connections if the service supports them. Measure resets at the client as well as successful exits at the server.
An interview answer should state which component stops admission and which deadline ultimately forces termination. “We added a sleep” is incomplete unless the sleep addresses a measured propagation delay within a tested shutdown sequence.
Self-check: all targets fail their health checks. Can the team assume the ALB will send no customer requests to them?
No. Account for ALB's documented fail-open behavior and design a separate controlled way to stop or redirect traffic when that is the desired action. Health status alone does not establish either graceful draining or complete traffic isolation.