RTO, RPO and recovery testing: prove service restoration and data loss limits
Define recovery time and recovery point objectives with a worked outage timeline. Test restored data, dependencies and business operations instead of relying on backup status.
TL;DR: RTO limits how long the service may remain unavailable. RPO limits the age of the recovery point relative to the disruption. Prove both with a restore exercise that checks business operations and identifies which writes survived.
Define the operation that must recover
A database accepting TCP connections is an incomplete recovery criterion for an order service. The application may still lack its encryption key, identity configuration, queue subscriptions or network route. Agree on an observable business operation, such as reading a known order and accepting a new idempotent order, before timing the exercise.
Recovery Time Objective is a target for the maximum interruption interval. Recovery Point Objective describes the tolerable data-loss window measured in time. AWS's disaster recovery objectives defines these separately. They are business decisions constrained by engineering cost, rather than properties automatically supplied by a backup schedule.
A backup every five minutes does not prove a five-minute RPO. Jobs can fail, replication can lag, copies can be inaccessible, and application data may be inconsistent across stores. The relevant point is the newest state you can actually recover and validate after the assumed failure.
Calculate two outcomes from one outage
Consider a hypothetical interruption beginning at 14:00. The latest verified recoverable transaction is timestamped 13:52. Engineers detect the outage at 14:04, decide to recover at 14:10, finish the database restore at 14:32, and restore the agreed business operation at 14:47.
With a 60-minute RTO and a five-minute RPO, the time target passes at 47 minutes. The recovery-point target fails because eight minutes of acknowledged activity may be missing. Reporting “database restored in 22 minutes” excludes detection, decision-making and service validation, so it does not measure the agreed interruption interval.
Transactions have relationships. If the order table recovers through 13:52 and the payment ledger through 13:58, the apparent newer payment copy can leave charges without matching orders. Define reconciliation rules and use stable transaction identifiers. Recovery can require replaying an external event source or investigating ambiguous writes rather than blindly replaying every request.
| Test evidence | What it establishes | Remaining limitation |
|---|---|---|
| Backup job succeeded | A job reported successful completion | Restore usability is unproven |
| Snapshot restored and opens | Storage is readable | Application consistency is unproven |
| Known transaction markers match | A specific recovery point survived | Other dependencies may still fail |
| Business operation passes | Service path works in the exercise | Failure scope and test conditions still matter |
Restore into an isolated destination
Keep the source intact during a rehearsal. Use a separate destination with controlled credentials and prevent restored workers from sending real payments, notifications or duplicate events. A restored system can retain production endpoints inside its configuration even when its compute is isolated.
Insert identifiable synthetic records in the exercise dataset at known points. After restore, compare the recovered marker with the planned point and run checks for referential integrity and expected counts. Counts alone are weak: two different sets of rows can have the same size. Include representative values or checksums where their semantics are meaningful.
A small local SQLite exercise can demonstrate this logic: write records 1 through 4, take a backup, add record 5, restore the backup into a new database and verify that record 5 is absent. That proves the snapshot boundary of the local example. It does not measure a managed database's failover time or establish the organization's RPO.
For an engine-version change, database upgrade rehearsals add driver and extension compatibility checks and a recovery boundary for writes accepted after cutover.
Include the dependencies that can block recovery
Document backup decryption, destination capacity, DNS changes and identity access. Assign an owner for the decision to fail over and a separate plan for returning to normal operation. A replica that immediately applies accidental deletion may be highly available while preserving no usable pre-deletion state. Keep the recovery strategy aligned with the failure being tested.
Self-check: a region-failure drill restores all database rows inside the target window, but the application cannot obtain a required key from the unavailable region. Has RTO passed? No, if that key is required for the agreed business operation. Include key availability in the recovery dependency map and retest the whole service. State explicitly which failure scope the improved exercise covers.