1Understand
The idea
Consensus is how a distributed system picks one writer, one lock holder, or one committed configuration when networks delay messages and partitions happen. Interviews want quorum math, leader leases, fencing, and judgment on buy vs build, not a full Raft derivation on the whiteboard.
Naive primary election (“heartbeat says I’m master”) without majority vote or storage fencing yields split brain: two primaries, corrupted ledger, angry finance. Managed RDS failover, etcd, and ZooKeeper exist because protocol bugs are expensive.
Consensus has a latency tax: quorum acks on every write; leader in a distant AZ adds RTT. Name what must be singular in *your* design before reaching for a coordination service.
Intuition
Why a leader. Many stores need single-writer serialization: SQL primary, distributed lock, global job runner. Multiple uncoordinated writers → lost updates unless you have CRDT/conflict rules (hard).
Quorum: N nodes, tolerate f failures if each decision needs majority ⌊N/2⌋+1. 3 nodes → 1 failure; 5 nodes → 2 failures. 2f+1 is the interview formula.
Leader election & term. Candidates request votes; higher term wins; leader sends heartbeats; stale leader must stop when term advances. Lease TTL: followers assume leader dead after missed heartbeats, too aggressive → flapping; too slow → long unavailability.
Fencing. Late leader must not write after new leader elected, fencing token monotonic with storage (e.g. SQL compares epoch before write). Locks without fencing: holder pauses, lease expires, new holder starts, old wakes and double-runs job.
Distributed lock smell. Redis SETNX + TTL alone is not consensus; use etcd lease, or DB row lock FOR UPDATE SKIP LOCKED on lease table with fencing id.
Buy managed: Spanner/RDS Multi-AZ, Cloud SQL failover, etcd for k8s, cite these for app interviews. DIY Raft only if designing coordination product.
Read path: followers may serve stale reads; leader serves linearizable writes depending on protocol.
Witness / observer nodes. Some clusters use non-voting observers for read scaling, mention only if interviewer goes deep; quorum still required for commits.
CAP framing (brief). During partition, choose explicit trade: stop writes to preserve consistency, or allow stale reads, tie to product, not poster slogans.
Lease duration tuning. Too short → flappy leadership; too long → long unavailability after real leader death, say you would tune with metrics on election rate.
Storage interaction. Leader epoch must be stored with writes, object storage and SQL primaries both need the fence token comparison, not only the lock service.
What you will be able to do
- Explain why naive “everyone is primary” fails
- Use quorum and leader lease at an interview level
- Know when to buy consensus (etcd, ZK, DB) vs hand-roll
When this comes up
- Leader election
- Quorum
- Split brain
- Raft / Paxos intuition
2See it
Architecture sketch
Quorum elects one leader
Majority vote picks who may write. Fencing tokens stop a stale leader after partition.
Walkthrough
Partition and fencing
Why “I still hear heartbeats” is not enough
1 / 4
Network split: old leader cannot reach a majority. Quorum elects a new leader.
Read the walkthrough as text
Worked example, exactly-once nightly billing job across 12 stateless API pods.
- Wrong:
@Scheduledon every pod, duplicate invoices. - Requirement: at most one runner; if leader dies, another resumes within minutes.
- Option A, etcd lease: key
/jobs/billing/leader; winner runs; holder renews every 10s; TTL 30s; fencing token passed tojob_runstable; storage rejects lower epoch. - Option B, SQL advisory lock: single-row
job_leasesupdated withWHERE lease_until < now(), consensus borrowed from existing primary. - Quorum cluster: if using 3-node etcd, writes need majority, mention failure tolerance 1 node.
- Split brain story: two pods think leader without quorum, both must fail to write job completion without winning fence.
- Do not whiteboard Paxos for shopping cart, “managed failover” suffices.
- Latency: election 5-30s window where writes pause, state expected unavailability.
What interviewers listen for: what is unique; quorum size; lease + fencing; managed vs DIY; split-brain prevention.
Napkin math
N nodes, quorum = floor(N/2) + 1
N=3 → need 2; N=5 → need 3
one leader for writes; followers replicate log
partition: old leader loses quorum, stop writes
fencing token / epoch blocks stale leader
Majority owns the truth. Fence zombies after split-brain.
3Answer it
How to answer in an interview
When single primary, lock, leader election, or “run once fleet-wide”:
- Uniqueness target, DB writer, lock, cron, config epoch.
- Failure tolerance,
ffailures →2f+1nodes if DIY coordination. - Leader duties, heartbeats, term/lease, re-election time.
- Split brain, fencing or quorum stop writes.
- Buy vs build, etcd, SQL lease, managed DB.
- Latency impact, quorum write RTT.
Phrases: “majority ack,” “fencing token,” “stale leader must not commit,” “borrow consensus from the primary DB.”
- Close, “I would buy consensus, not become a consensus company.”
Answer checklist
Tradeoffs & numbers
- Strong leader: simple writes; election blip on failure.
- Quorum every write: durable; slower than async replicate.
- DIY Raft/ZK: control; years of edge cases, avoid as default app answer.
- Redis lock only: fast; split-brain without fencing.
- Multi-primary DB: needs conflict resolution, say CRDT or last-write-wins cost.
- When NOT consensus layer: stateless API tier; idempotent workers with external idempotency store.
- Observer cost: extra ZK/etcd nodes for reads only, mention if interviewer asks about read scaling control plane.
- Interview tip: cite
2f+1with a concrete N (5 nodes, tolerate 2 failures), interviewers want a number. - Real systems: cite managed Postgres failover before drawing Paxos, credibility beats protocol theater.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Single primary election
Example. Postgres primary dies; only one writer must accept orders. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.
How. Managed failover promotes sync replica; apps reconnect to new primary DNS; brief write pause. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.
Quorum writes
Example. Distributed config store: change must survive loss of two nodes in five-node cluster. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.
How. Leader proposes; majority append before ack to client; minority partition stops accepting writes. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.
Distributed lock
Example. Fleet-wide mutex for tax filing job on April 1. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.
How. etcd lease lock + fencing counter passed to output file writer; stale holder cannot overwrite. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.
Split-brain scenario
Example. AZ partition isolates old primary from quorum but it still accepts traffic. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.
How. STONITH/fencing to storage; clients only talk to node with majority; reject dual write story. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Election without fencing
Avoid. Heartbeat says “I’m primary.” Two primaries after a partition.
Prefer. Quorum election + fencing token on every write path.
Weak sketch
Bad
Stronger sketch
Good
Two primaries
Why it happens. Uncoordinated dual writers produce divergent state that async sync cannot safely merge for financial data.
How it fails. Active-active MySQL primaries in US and EU both take writes with “conflict resolution later.”
Fix. Single leader per shard with quorum/fencing; or explicit CRDT/product rules for conflicts.
Ignores quorum
Why it happens. Committing without majority ack loses durability under partition, minority leader can acknowledge alone.
How it fails. Leader returns write success after persisting locally while split from 3 of 5 replicas.
Fix. Quorum commit; minority partition stops serving writes until healed.
DIY Raft
Why it happens. Protocol edge cases (log holes, joint consensus) burn engineering years; interviews test product judgment.
How it fails. Shopping cart design spends 20 minutes inventing Paxos for coupon counters.
Fix. Managed DB failover, etcd lease, or SQL `SKIP LOCKED` lease; mention quorum without deriving Raft.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 9 min
Single cron leader
Ten stateless API pods each run a midnight billing cron. Jobs sometimes run twice, double-charging customers. Design leader election so exactly one pod runs the job, including what happens when the leader dies mid-run.
- about 8 min
Quorum size
A 5-node coordination cluster must tolerate two simultaneous node failures without losing committed configuration changes. How many nodes must ack a write, and what happens to a minority partition during a network split?
- about 8 min
Distributed lock
Workers must not process the same invoice PDF generation concurrently. A developer uses Redis `SETNX` with a 30s TTL and no fencing. What failure modes remain, and how would you harden the lock?
- about 9 min
Split brain
A network partition isolates two database nodes; both promote themselves primary and accept writes for 90 seconds until the partition heals. How do you prevent this in design, and what fencing or quorum rules would you cite?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.