Fundamentals

Availability & Replication

Say what “up” means, how replicas help, and what you give up when you prefer availability over consistency.

Intermediate, about 70 minutes to read, 4 open questions to practise.

1Understand

The idea

Availability is whether the product stays usable when parts break: a disk, a host, a rack, or a whole availability zone. Replication is not decoration: it is how you remove single points of failure and bound how long recovery takes. In interviews, “three nines” only matters after you say *what fails* and *what users still see* during and after that failure.

High availability almost always trades against consistency and complexity. Followers can serve stale reads; failover can take seconds; multi-region adds latency and conflict stories. Strong answers name the failure domain, who takes writes after failover, and which clients might read slightly old data, not a CAP poster slogan.

Tie availability to a user-journey SLO (“99.9% of checkouts succeed monthly”) and to tested failover: health checks, fencing, and runbooks, not a standby box nobody promotes.

Intuition

What “up” means. Define the user-visible failure: “users can still place orders” vs “dashboard loads.” A database that accepts writes but cannot serve reads is a partial outage; say which SLO you are defending.

SPOF hunting. Walk the request path: load balancer, app tier, single primary DB, one Redis, one queue broker. Each singleton without a failover story is a bet the interview will probe.

Replication roles. A leader (primary) takes writes; followers (replicas) apply the log and serve reads or stand by for promotion. Read replicas scale read traffic but replicate asynchronously, followers can lag, so “read your own write” may require routing recent writes to the leader or a sync wait.

Failover. On primary death: automatic promotion (managed DB), election (consensus), or manual runbook. Name RTO (how long until service returns) and RPO (how much data you might lose if the last seconds of writes did not replicate). Sync replication tightens RPO; async loosens it but keeps latency down.

Multi-AZ vs multi-region. Multi-AZ survives a datacenter fault with milliseconds-seconds of failover inside one region. Multi-region targets regional disasters and often means active-passive (one writes, DR site catches up) or active-active (harder writes/conflicts). Stale reads during partition are the concrete cost of favoring availability.

Degraded mode: payment provider down → queue or banner, not infinite spinner. Blast radius: AZ loss should not equal global outage if architecture is cell-aware.

Chaos drills: periodic failover exercises validate RTO; mention once in senior answers.

What you will be able to do

  • Name single points of failure and how replication removes them
  • Explain leader/follower (or multi-leader) for writes
  • Trade availability vs consistency without hand-waving CAP

When this comes up

  • Single point of failure
  • Multi-AZ / multi-region
  • RPO / RTO
  • Leader vs follower

2See it

Architecture sketch

Primary + replicas

Writes hit the primary. Replicas copy the log. Reads may go to a replica and see lag.

100%
ServiceStore

Walkthrough

Primary fails over

Who takes writes?

1 / 4

100%

Healthy: writes to primary, optional reads on replicas (may be stale).

Read the walkthrough as text

Case, orders API with Postgres primary.

Prompt: payment capture must survive single host failure; reads can be slightly stale on catalog pages.

  1. Failure domain: single AZ loss, not “the internet.”
  2. SPOF today: one Postgres primary in one AZ → fix: multi-AZ managed Postgres with synchronous replica in another AZ for automatic failover (typical RTO under a minute, RPO near zero for that sync pair).
  3. Reads: hot product catalog → read replicas or read endpoint with “eventual” lag called out; checkout must read inventory from leader or use version checks.
  4. App tier: N instances behind LB, already no SPOF at app layer.
  5. Consistency story: “After adding to cart, user might see old stock count on a replica for a second; at payment we read authoritative stock from primary.”
  6. **What you do *not* claim:** “We are AP so consistency does not matter.” You named *which* reads can lag.

Multi-region follow-up. If the business requires surviving region loss: async replica in second region, active-passive, promote on disaster with larger RPO; traffic steering via DNS/global LB. Say writes stay in primary region normally to avoid split-brain.

  1. Broker HA, if email queue is down, checkout may succeed but receipts stall; partial outage counts.
  2. LB tie-in, unhealthy app drains; clients retry to healthy nodes.

Phrase: “Catalog OK on replica with ~1s lag; inventory at payment reads primary.”

Napkin math

Nines napkin

99.9% ≈ 43 min downtime / month

99.99% ≈ 4.3 min / month

RPO = how much data you may lose

RTO = how fast you must recover

writes → primary; reads → replicas (lag)

failover: promote + fence old primary

State the RPO/RTO before drawing replicas.

3Answer it

How to answer in an interview

When the prompt mentions downtime, disasters, or “always on”:

  1. Define failure, node, AZ, region; what user journey must survive.
  2. Availability target, translate nines to downtime per month if helpful (99.9% ≈ 43 min/month).
  3. Find SPOFs on the diagram; add replicas/failover for each critical singleton.
  4. Write path, single leader vs multi-leader; who promotes on failure.
  5. Read path, replicas OK? stale window?
  6. RPO/RTO, rough, tied to sync vs async replication.
  7. Consistency cost, concrete stale-read or conflict examples, not CAP jargon alone.

Draw one primary with at least one follower before you talk about global scale.

Close with numbers: “RTO ~60s auto-failover, RPO ~0 on sync AZ pair for payments; CMS reads can lag minutes on replica.”

Answer checklist

1. Define the failure you care about (node, AZ, region). 2. Say the availability target (e.g. 99.9%) in plain language. 3. Place replicas / failover and who takes writes. 4. Name the consistency cost (stale reads, split brain). 5. State RPO/RTO roughly if durability matters.

Tradeoffs & numbers

  • Sync replication: smaller RPO, higher write latency and cross-AZ cost.
  • Async replication: faster writes; failover may lose the last second of writes, say if that is acceptable.
  • Read replicas: cheap read scale; lag breaks strict session consistency unless you route carefully.
  • Automatic failover: faster RTO; risk of false promotion (split brain) if health checks are wrong, fencing matters.
  • Multi-region active-active: best availability globally; conflict resolution and write latency dominate, only when the product demands it.
  • Cost: every extra AZ/region is ongoing infra; justify with the stated RTO/RPO, not fear.

When NOT multi-region: single-market product, multi-AZ often enough; multi-region adds conflict + latency. - Nines: 99.99% ≈ 4.3 min/month budget. Name what burns it. - Minority partition: fail closed on writes without quorum to avoid split-brain.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Single-node SPOF

    Example. One Postgres on a single VM serves all writes and reads. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).

    How. Call the primary a SPOF; add follower + automatic or manual failover before claiming high availability. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.

  • Read replicas

    Example. Dashboard and catalog reads dominate; writes stay on primary. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).

    How. Route heavy reads to replicas; state replication lag; send authoritative reads (balance, inventory) to leader. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.

  • Multi-AZ failover

    Example. Survive loss of one availability zone without manual restore. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).

    How. Sync/async replica in second AZ; health checks promote; name RTO/RPO and split-brain guards. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.

  • Multi-region active-passive

    Example. Region-wide disaster should not lose the product permanently. Name the failure domain (host, AZ, region), rough RTO/RPO, and one user journey that must stay up during failover (checkout, login, feed read).

    How. Primary region writes; async DR replica; DNS/LB traffic shift on disaster; accept larger RPO on promotion. Show primary + replica; who takes writes after promotion; which reads may be stale and for how long.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Replicas without a plan

Avoid. Draw replicas but never say who becomes primary or how clients find it.

Prefer. Named failover, fencing, and read-staleness expectations.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Ignores SPOF

    Why it happens. One database box with no replica is a scheduled outage when hardware fails.

    How it fails. Architecture diagram shows a single “Database” rectangle and “we scale horizontally” at the app tier only.

    Fix. Add at least one follower or managed multi-AZ primary; say who serves writes after failover.

  • CAP slogan

    Why it happens. “We choose AP” does not tell the interviewer which reads are stale or for how long.

    How it fails. Mentions CAP theorem without saying catalog reads can lag 500ms on a replica after a price update.

    Fix. Give a concrete user-visible example of staleness or a write that must read from the leader.

  • No failover

    Why it happens. Replicas you never promote still mean downtime while someone restores manually.

    How it fails. “We have a standby replica” with no automatic promotion: health checks, or RTO estimate.

    Fix. Describe promotion trigger, fencing/split-brain avoidance, and rough RTO/RPO.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons