Fundamentals

Consensus & Leader Election

Agree on a single leader or value across nodes when failure and partitions happen.

Advanced, about 75 minutes to read, 4 open questions to practise.

1Understand

The idea

Consensus is how a distributed system picks one writer, one lock holder, or one committed configuration when networks delay messages and partitions happen. Interviews want quorum math, leader leases, fencing, and judgment on buy vs build, not a full Raft derivation on the whiteboard.

Naive primary election (“heartbeat says I’m master”) without majority vote or storage fencing yields split brain: two primaries, corrupted ledger, angry finance. Managed RDS failover, etcd, and ZooKeeper exist because protocol bugs are expensive.

Consensus has a latency tax: quorum acks on every write; leader in a distant AZ adds RTT. Name what must be singular in *your* design before reaching for a coordination service.

Intuition

Why a leader. Many stores need single-writer serialization: SQL primary, distributed lock, global job runner. Multiple uncoordinated writers → lost updates unless you have CRDT/conflict rules (hard).

Quorum: N nodes, tolerate f failures if each decision needs majority ⌊N/2⌋+1. 3 nodes → 1 failure; 5 nodes → 2 failures. 2f+1 is the interview formula.

Leader election & term. Candidates request votes; higher term wins; leader sends heartbeats; stale leader must stop when term advances. Lease TTL: followers assume leader dead after missed heartbeats, too aggressive → flapping; too slow → long unavailability.

Fencing. Late leader must not write after new leader elected, fencing token monotonic with storage (e.g. SQL compares epoch before write). Locks without fencing: holder pauses, lease expires, new holder starts, old wakes and double-runs job.

Distributed lock smell. Redis SETNX + TTL alone is not consensus; use etcd lease, or DB row lock FOR UPDATE SKIP LOCKED on lease table with fencing id.

Buy managed: Spanner/RDS Multi-AZ, Cloud SQL failover, etcd for k8s, cite these for app interviews. DIY Raft only if designing coordination product.

Read path: followers may serve stale reads; leader serves linearizable writes depending on protocol.

Witness / observer nodes. Some clusters use non-voting observers for read scaling, mention only if interviewer goes deep; quorum still required for commits.

CAP framing (brief). During partition, choose explicit trade: stop writes to preserve consistency, or allow stale reads, tie to product, not poster slogans.

Lease duration tuning. Too short → flappy leadership; too long → long unavailability after real leader death, say you would tune with metrics on election rate.

Storage interaction. Leader epoch must be stored with writes, object storage and SQL primaries both need the fence token comparison, not only the lock service.

What you will be able to do

  • Explain why naive “everyone is primary” fails
  • Use quorum and leader lease at an interview level
  • Know when to buy consensus (etcd, ZK, DB) vs hand-roll

When this comes up

  • Leader election
  • Quorum
  • Split brain
  • Raft / Paxos intuition

2See it

Architecture sketch

Quorum elects one leader

Majority vote picks who may write. Fencing tokens stop a stale leader after partition.

100%
ClientServiceStore

Walkthrough

Partition and fencing

Why “I still hear heartbeats” is not enough

1 / 4

100%

Network split: old leader cannot reach a majority. Quorum elects a new leader.

Read the walkthrough as text

Worked example, exactly-once nightly billing job across 12 stateless API pods.

  1. Wrong: @Scheduled on every pod, duplicate invoices.
  2. Requirement: at most one runner; if leader dies, another resumes within minutes.
  3. Option A, etcd lease: key /jobs/billing/leader; winner runs; holder renews every 10s; TTL 30s; fencing token passed to job_runs table; storage rejects lower epoch.
  4. Option B, SQL advisory lock: single-row job_leases updated with WHERE lease_until < now(), consensus borrowed from existing primary.
  5. Quorum cluster: if using 3-node etcd, writes need majority, mention failure tolerance 1 node.
  6. Split brain story: two pods think leader without quorum, both must fail to write job completion without winning fence.
  7. Do not whiteboard Paxos for shopping cart, “managed failover” suffices.
  8. Latency: election 5-30s window where writes pause, state expected unavailability.

What interviewers listen for: what is unique; quorum size; lease + fencing; managed vs DIY; split-brain prevention.

Napkin math

Quorum napkin

N nodes, quorum = floor(N/2) + 1

N=3 → need 2; N=5 → need 3

one leader for writes; followers replicate log

partition: old leader loses quorum, stop writes

fencing token / epoch blocks stale leader

Majority owns the truth. Fence zombies after split-brain.

3Answer it

How to answer in an interview

When single primary, lock, leader election, or “run once fleet-wide”:

  1. Uniqueness target, DB writer, lock, cron, config epoch.
  2. Failure tolerance, f failures → 2f+1 nodes if DIY coordination.
  3. Leader duties, heartbeats, term/lease, re-election time.
  4. Split brain, fencing or quorum stop writes.
  5. Buy vs build, etcd, SQL lease, managed DB.
  6. Latency impact, quorum write RTT.

Phrases: “majority ack,” “fencing token,” “stale leader must not commit,” “borrow consensus from the primary DB.”

  1. Close, “I would buy consensus, not become a consensus company.”

Answer checklist

1. What must be unique (primary, lock, config)? 2. Quorum size and failure tolerance. 3. Leader duties and lease / term. 4. Split-brain prevention. 5. When not to invent consensus (use managed lock / DB).

Tradeoffs & numbers

  • Strong leader: simple writes; election blip on failure.
  • Quorum every write: durable; slower than async replicate.
  • DIY Raft/ZK: control; years of edge cases, avoid as default app answer.
  • Redis lock only: fast; split-brain without fencing.
  • Multi-primary DB: needs conflict resolution, say CRDT or last-write-wins cost.
  • When NOT consensus layer: stateless API tier; idempotent workers with external idempotency store.
  • Observer cost: extra ZK/etcd nodes for reads only, mention if interviewer asks about read scaling control plane.
  • Interview tip: cite 2f+1 with a concrete N (5 nodes, tolerate 2 failures), interviewers want a number.
  • Real systems: cite managed Postgres failover before drawing Paxos, credibility beats protocol theater.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Single primary election

    Example. Postgres primary dies; only one writer must accept orders. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.

    How. Managed failover promotes sync replica; apps reconnect to new primary DNS; brief write pause. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.

  • Quorum writes

    Example. Distributed config store: change must survive loss of two nodes in five-node cluster. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.

    How. Leader proposes; majority append before ack to client; minority partition stops accepting writes. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.

  • Distributed lock

    Example. Fleet-wide mutex for tax filing job on April 1. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.

    How. etcd lease lock + fencing counter passed to output file writer; stale holder cannot overwrite. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.

  • Split-brain scenario

    Example. AZ partition isolates old primary from quorum but it still accepts traffic. Give quorum math, leader lease/fencing, and when managed coordination beats a homegrown election on the critical path.

    How. STONITH/fencing to storage; clients only talk to node with majority; reject dual write story. Odd member count; lease TTL; fencing token on storage writes; buy etcd/ZK when split brain is unacceptable.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Election without fencing

Avoid. Heartbeat says “I’m primary.” Two primaries after a partition.

Prefer. Quorum election + fencing token on every write path.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Two primaries

    Why it happens. Uncoordinated dual writers produce divergent state that async sync cannot safely merge for financial data.

    How it fails. Active-active MySQL primaries in US and EU both take writes with “conflict resolution later.”

    Fix. Single leader per shard with quorum/fencing; or explicit CRDT/product rules for conflicts.

  • Ignores quorum

    Why it happens. Committing without majority ack loses durability under partition, minority leader can acknowledge alone.

    How it fails. Leader returns write success after persisting locally while split from 3 of 5 replicas.

    Fix. Quorum commit; minority partition stops serving writes until healed.

  • DIY Raft

    Why it happens. Protocol edge cases (log holes, joint consensus) burn engineering years; interviews test product judgment.

    How it fails. Shopping cart design spends 20 minutes inventing Paxos for coupon counters.

    Fix. Managed DB failover, etcd lease, or SQL `SKIP LOCKED` lease; mention quorum without deriving Raft.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons