Managed database failover: promotion, DNS and application recovery
Explain RDS failover beyond replica promotion. Follow endpoint changes, stale connections and ambiguous commits, then define an application-level recovery test.
TL;DR: A promoted database is useful when the application can reconnect and safely resume work. Test writer selection, endpoint resolution, stale connection removal and uncertain transaction recovery as one service path.
Name the deployment before promising its behavior
“RDS has a standby” leaves several architectures unresolved. A traditional Multi-AZ DB instance has a synchronous standby used for availability; that standby is not an ordinary read-scaling endpoint. RDS Multi-AZ DB clusters and Aurora have different reader and failover arrangements. A conventional asynchronous read replica has another purpose and can require a separate promotion procedure.
AWS's Multi-AZ overview distinguishes these deployment forms. Ask which engine and topology are running before describing an automatic promotion path or data-loss expectation. A replica in another zone also does not replace point-in-time recovery for an accidental deletion that replication faithfully copies.
The following explanation uses an RDS Multi-AZ DB instance. Its failover documentation describes endpoint DNS changes and client reconnection considerations. Treat documented timing ranges as expectations to test under your workload, not a contractual bound on application recovery.
Separate the database event from the customer interruption
An existing TCP connection does not migrate to the new database process when DNS changes. The pool must discard failed sessions and create new ones. The new connection may still resolve an old address if the runtime caches DNS longer than intended. Inspect the actual driver, resolver and pool settings together. A healthy DNS command in a shell does not establish what a long-running JVM has cached.
Consider a fictional failover drill. The writer fails at 10:00:00 and a replacement is usable at 10:00:45. The application's first fresh connection succeeds at 10:01:10, but its checkout operation does not recover until 10:01:25 because workers keep retrying a failed transaction incorrectly. The observed interruption is 85 seconds. Reporting only the 45-second promotion conceals 40 seconds of client recovery. These values are illustrative, not AWS measurements.
| Layer | Recovery evidence | Failure that can outlive promotion |
|---|---|---|
| Managed service | New writer accepts the required operation | Promotion or recovery still incomplete |
| Resolver | Application resolves the current endpoint | Runtime DNS cache retains old address |
| Connection pool | Failed sessions leave the pool | Dead sockets repeatedly handed out |
| Transaction handling | Business effect is known or reconciled | Commit result was lost |
A lost response makes commit outcome uncertain
Suppose the database commits an order, then the connection closes before the client receives confirmation. A retry that generates a new order ID can produce a second order. Conversely, treating every connection failure as successful loses work when the transaction never committed.
Use a stable operation ID established before the first attempt. The authoritative database record should let a later attempt discover whether that operation already committed. The idempotency concept explains the transactional boundary; a process-local cache cannot settle the outcome after that process restarts.
Keep recovery bounded by the caller's deadline. Connection retries need backoff and a concurrency limit so every application replica does not rebuild its entire pool at once. Reconnect and transaction retry are separate decisions: opening a new session is safe in circumstances where replaying a business operation requires reconciliation first.
Rehearse the application's failure path
Use an authorized non-production failover exercise with synthetic operation IDs and a known client load. Record the first failed operation, service events, application resolver observations, pool errors and the first sustained successful business operation. Include a transaction near the disruption boundary and verify the final count of committed effects by ID.
A proxy can reduce some connection-management work, but its transaction and session behavior still matters. AWS's RDS Proxy pinning guidance describes cases where client state ties a session to a backend. Do not assume a proxy makes every session interchangeable or transparently replays an interrupted transaction.
Test low traffic as well as a busy pool. A service with little traffic can retain idle connections for a long time, while a busy service can create a reconnection surge. Check read-only behavior, authentication and encryption on the replacement path; successful TCP connection alone is insufficient.
Self-check: the provider reports failover complete and new test connections work, but one application instance still fails every request. What do you compare first? Its resolved address, connection-pool disposal behavior and driver errors against a healthy instance. Preserve the evidence before restarting it. A restart might restore service, but only those comparisons explain which client setting needs correction.