1Understand
The idea
Availability is whether the product stays usable when parts break: a disk, a host, a rack, or a whole availability zone. Replication is not decoration: it is how you remove single points of failure and bound how long recovery takes. In interviews, “three nines” only matters after you say *what fails* and *what users still see* during and after that failure.
High availability almost always trades against consistency and complexity. Followers can serve stale reads; failover can take seconds; multi-region adds latency and conflict stories. Strong answers name the failure domain, who takes writes after failover, and which clients might read slightly old data, not a CAP poster slogan.
Tie availability to a user-journey SLO (“99.9% of checkouts succeed monthly”) and to tested failover: health checks, fencing, and runbooks, not a standby box nobody promotes.
Intuition
What “up” means. Define the user-visible failure: “users can still place orders” vs “dashboard loads.” A database that accepts writes but cannot serve reads is a partial outage; say which SLO you are defending.
SPOF hunting. Walk the request path: load balancer, app tier, single primary DB, one Redis, one queue broker. Each singleton without a failover story is a bet the interview will probe.
Replication roles. A leader (primary) takes writes; followers (replicas) apply the log and serve reads or stand by for promotion. Read replicas scale read traffic but replicate asynchronously, followers can lag, so “read your own write” may require routing recent writes to the leader or a sync wait.
Failover. On primary death: automatic promotion (managed DB), election (consensus), or manual runbook. Name RTO (how long until service returns) and RPO (how much data you might lose if the last seconds of writes did not replicate). Sync replication tightens RPO; async loosens it but keeps latency down.
Multi-AZ vs multi-region. Multi-AZ survives a datacenter fault with milliseconds-seconds of failover inside one region. Multi-region targets regional disasters and often means active-passive (one writes, DR site catches up) or active-active (harder writes/conflicts). Stale reads during partition are the concrete cost of favoring availability.
Degraded mode: payment provider down → queue or banner, not infinite spinner. Blast radius: AZ loss should not equal global outage if architecture is cell-aware.
Chaos drills: periodic failover exercises validate RTO; mention once in senior answers.
What you will be able to do
- Name single points of failure and how replication removes them
- Explain leader/follower (or multi-leader) for writes
- Trade availability vs consistency without hand-waving CAP
When this comes up
- Single point of failure
- Multi-AZ / multi-region
- RPO / RTO
- Leader vs follower
2See it
Architecture sketch
Primary + replicas
Writes hit the primary. Replicas copy the log. Reads may go to a replica and see lag.
Walkthrough
Primary fails over
Who takes writes?
1 / 4
Healthy: writes to primary, optional reads on replicas (may be stale).
Read the walkthrough as text
Case, orders API with Postgres primary.
Prompt: payment capture must survive single host failure; reads can be slightly stale on catalog pages.
- Failure domain: single AZ loss, not “the internet.”
- SPOF today: one Postgres primary in one AZ → fix: multi-AZ managed Postgres with synchronous replica in another AZ for automatic failover (typical RTO under a minute, RPO near zero for that sync pair).
- Reads: hot product catalog → read replicas or read endpoint with “eventual” lag called out; checkout must read inventory from leader or use version checks.
- App tier: N instances behind LB, already no SPOF at app layer.
- Consistency story: “After adding to cart, user might see old stock count on a replica for a second; at payment we read authoritative stock from primary.”
- **What you do *not* claim:** “We are AP so consistency does not matter.” You named *which* reads can lag.
Multi-region follow-up. If the business requires surviving region loss: async replica in second region, active-passive, promote on disaster with larger RPO; traffic steering via DNS/global LB. Say writes stay in primary region normally to avoid split-brain.
- Broker HA, if email queue is down, checkout may succeed but receipts stall; partial outage counts.
- LB tie-in, unhealthy app drains; clients retry to healthy nodes.
Phrase: “Catalog OK on replica with ~1s lag; inventory at payment reads primary.”
Napkin math
99.9% ≈ 43 min downtime / month
99.99% ≈ 4.3 min / month
RPO = how much data you may lose
RTO = how fast you must recover
writes → primary; reads → replicas (lag)
failover: promote + fence old primary
State the RPO/RTO before drawing replicas.
3Answer it
How to answer in an interview
When the prompt mentions downtime, disasters, or “always on”:
- Define failure, node, AZ, region; what user journey must survive.
- Availability target, translate nines to downtime per month if helpful (99.9% ≈ 43 min/month).
- Find SPOFs on the diagram; add replicas/failover for each critical singleton.
- Write path, single leader vs multi-leader; who promotes on failure.
- Read path, replicas OK? stale window?
- RPO/RTO, rough, tied to sync vs async replication.
- Consistency cost, concrete stale-read or conflict examples, not CAP jargon alone.
Draw one primary with at least one follower before you talk about global scale.
Close with numbers: “RTO ~60s auto-failover, RPO ~0 on sync AZ pair for payments; CMS reads can lag minutes on replica.”
Answer checklist
Tradeoffs & numbers
- Sync replication: smaller RPO, higher write latency and cross-AZ cost.
- Async replication: faster writes; failover may lose the last second of writes, say if that is acceptable.
- Read replicas: cheap read scale; lag breaks strict session consistency unless you route carefully.
- Automatic failover: faster RTO; risk of false promotion (split brain) if health checks are wrong, fencing matters.
- Multi-region active-active: best availability globally; conflict resolution and write latency dominate, only when the product demands it.
- Cost: every extra AZ/region is ongoing infra; justify with the stated RTO/RPO, not fear.
When NOT multi-region: single-market product, multi-AZ often enough; multi-region adds conflict + latency. - Nines: 99.99% ≈ 4.3 min/month budget. Name what burns it. - Minority partition: fail closed on writes without quorum to avoid split-brain.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Single-node SPOF
Example. One Postgres on a single VM serves all writes and reads. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).
How. Call the primary a SPOF; add follower + automatic or manual failover before claiming high availability. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.
Read replicas
Example. Dashboard and catalog reads dominate; writes stay on primary. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).
How. Route heavy reads to replicas; state replication lag; send authoritative reads (balance, inventory) to leader. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.
Multi-AZ failover
Example. Survive loss of one availability zone without manual restore. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).
How. Sync/async replica in second AZ; health checks promote; name RTO/RPO and split-brain guards. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.
Multi-region active-passive
Example. Region-wide disaster should not lose the product permanently. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).
How. Primary region writes; async DR replica; DNS/LB traffic shift on disaster; accept larger RPO on promotion. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Replicas without a plan
Avoid. Draw replicas but never say who becomes primary or how clients find it.
Prefer. Named failover, fencing, and read-staleness expectations.
Weak sketch
Bad
Stronger sketch
Good
Ignores SPOF
Why it happens. One database box with no replica is a scheduled outage when hardware fails.
How it fails. Architecture diagram shows a single “Database” rectangle and “we scale horizontally” at the app tier only.
Fix. Add at least one follower or managed multi-AZ primary; say who serves writes after failover.
CAP slogan
Why it happens. “We choose AP” does not tell the interviewer which reads are stale or for how long.
How it fails. Mentions CAP theorem without saying catalog reads can lag 500ms on a replica after a price update.
Fix. Give a concrete user-visible example of staleness or a write that must read from the leader.
No failover
Why it happens. Replicas you never promote still mean downtime while someone restores manually.
How it fails. “We have a standby replica” with no automatic promotion: health checks, or RTO estimate.
Fix. Describe promotion trigger, fencing/split-brain avoidance, and rough RTO/RPO.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Find the SPOF
A sketch shows clients → load balancer → 8 stateless API pods → one Postgres primary on a single host (no replicas). The PM asks for 99.9% availability. In 6-10 sentences, what fails first, what replication/failover you would add, and what consistency tradeoff you accept on reads.
- about 8 min
Read replica routing
90% of traffic is read-only product catalog queries; 10% is checkout writes. You add Postgres read replicas. How do you route reads and writes? When can a user see stale data, and what do you do for “read your own write” on the cart page?
- about 9 min
Multi-AZ failover
Your service must survive the loss of one AWS availability zone with minimal manual ops. Describe Postgres (or managed SQL) layout across AZs, failover behavior, and realistic RTO/RPO. What prevents split brain during promotion?
- about 10 min
Active-passive region DR
Leadership wants disaster recovery if an entire region is unavailable for hours. Outline an active-passive multi-region design for a write-heavy API: where writes go normally, how DR catches up, how you fail over traffic, and what data loss window you admit.
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.