DevOpsInterviewPrep logo
DevOps System Design & Architecture / 05
hard★ EssentialNewAmazonRazorpaySnowflake

Make a Postgres database highly available. What are you actually promising, and what breaks?

Every candidate says replication and failover. The scoring answer names the replication mode, states the resulting data loss window, and explains how the cluster avoids two primaries.

Updated Sep 2026 · Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Availability for a database is a promise about failover time and data loss, so state both. Asynchronous replication gives fast failover with a data-loss window equal to replication lag; synchronous durable acknowledgement plus safe promotion can protect acknowledged commits and puts a round trip inside every commit. Whatever you choose, the hard part is fencing: preventing two nodes believing they are primary.

How to approach it

Refuse to answer without RTO and RPO, because they determine the design. Then pick a replication mode, state its cost honestly, and spend the rest on failover safety, which is where these systems actually fail.

A strong answer

Start with what is being promised. RTO is how long you may be down; RPO is how much data you may lose. "Highly available" with neither number is not a requirement.

Replication mode is the first decision. Asynchronous replication means the primary commits and ships changes to replicas without waiting. Writes stay fast, and if the primary dies, acknowledged WAL unavailable on the promoted survivor may be lost. Distinguish sent, written, flushed and replayed WAL; replay lag alone can overstate loss when the WAL is already durable on that survivor. That makes lag a first-class monitored, alerted signal rather than a dashboard curiosity, which is the detail that separates people who have run this from people who have read about it.

Synchronous replication waits for at least one replica to confirm before acknowledging the commit. With synchronous_commit=on, acknowledgement waits for WAL flush on the required synchronous standby set. Protecting acknowledged commits also requires promoting a survivor containing that WAL and fencing the old primary; remote_write alone is not the same durability promise. Within an availability zone that is sub-millisecond and usually acceptable; across regions it is tens of milliseconds on every commit and usually is not. Postgres ANY quorum and FIRST priority rules choose which synchronous standbys acknowledge; quorum is a synchronous selection policy, not a middle mode between asynchronous and synchronous replication.

rendering diagram…

Failover is where these designs break. Automatic failover needs something to decide the primary is dead, and a network partition looks identical to a dead node from one side. If both sides promote, you have two primaries taking writes and divergent data that no later reconciliation fully repairs. So you need consensus rather than a timer: an odd number of voting members, a quorum requirement, and fencing that makes the old primary stop accepting writes even if it comes back. Patroni with etcd or Consul is the standard open-source answer; managed services implement the equivalent internally.

Then the parts people forget. Applications must reconnect to the new primary, which means a virtual IP, a proxy such as PgBouncer or HAProxy, or DNS with a short TTL, and connection pools must actually drop and re-establish rather than holding dead sockets. Read replicas serve reads but are behind, so an application that writes then immediately reads may not see its own write, and that is a correctness question for the application rather than the database.

And replication is not backup. It faithfully replicates a bad migration, a mistaken delete and a ransomware encryption. You still need point-in-time recovery with tested restores, because the failure you are most likely to have is a human one.

What interviewers probe next

"What is your actual RPO on async?" Compare committed WAL with durable WAL on eligible promotion targets. Replay lag is a recovery-time input as well; monitor the distinction against the promised RPO.

"Managed or self-hosted?" Managed for almost everyone: failover, backups and patching are the hard parts and the operational burden is the real cost. Self-host when you need an extension or a version the provider does not offer, and be honest that you are taking on a specialist load.

"How do you test it?" Scheduled failover in business hours against real traffic. An untested failover has an unknown success probability.

Common mistakes

Saying "replication and automatic failover" without naming the mode or the resulting RPO.

Ignoring split brain, which is the failure mode that turns an outage into data corruption.

Treating replicas as backups, which leaves you with no recovery path from a logical error.

References

PostgreSQL standby replication.

That one was free, and so are 10 answers per topic without an account. Signing in doubles that to 20, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
DISCUSSION · 0

Nothing here yet. Say how you would answer it.