Caching & Scale

Load Balancing

Spread traffic across healthy instances and remove single servers from the critical path.

Beginner, about 55 minutes to read, 4 open questions to practise.

1Understand

The idea

A load balancer spreads incoming connections and requests across a pool of healthy backends so no single instance carries the whole product. In interviews it is the glue between “we run N stateless APIs” and “clients actually hit one hostname.” The answer is not just “add an LB”, it is L4 vs L7, which algorithm fits the workload, health checks that stop sending traffic to dead nodes, and whether stickiness is required or a smell that the app is not stateless.

The LB itself can become a single point of failure, so managed load balancers, active/passive pairs, or Anycast matter. Good designs make session state live in a shared store or disappear entirely so round-robin stays fair.

Intuition

What is being balanced. HTTP/HTTPS APIs (L7), raw TCP (L4), gRPC streams, WebSockets. L4 LB is fast and connection-oriented; L7 can route by path, host header, cookies, or JWT claims.

Algorithms:

  • Round-robin, fair for homogeneous short requests on stateless instances.
  • Least connections, better when requests hold connections a long time (SSE, uploads, slow queries).
  • Consistent hash, sticky by key at the LB for cacheable partitions; overlaps with the consistent-hashing pattern but at the edge.
  • Weighted, canary or heterogeneous instance sizes.

Health checks. Active HTTP GET /health or TCP probe; unhealthy backends are drained, no new connections, in-flight requests may complete. Say graceful shutdown: app stops accepting, finishes work, then deregisters.

Statelessness vs stickiness. Ideal: any instance can serve any request; session in Redis/JWT. Sticky sessions (cookie to same backend) help legacy servers with in-memory session but hurt failover and uneven load, prefer shared session store.

LB HA. One hardware LB is a SPOF; use cloud managed LB, two LBs with floating IP, or Anycast anycast VIP.

What you will be able to do

  • Place an LB in front of a replicated stateless tier
  • Choose L4 vs L7 and a balancing algorithm with a reason
  • Explain health checks and draining

When this comes up

  • Stateless API tier
  • Health checks
  • L4 vs L7
  • Sticky sessions

2See it

Architecture sketch

Stateless API behind a load balancer

LB spreads traffic; health checks remove bad instances. Make apps stateless or share session state.

100%
ClientGatewayServiceStore

Walkthrough

Health check drain

Instance dies, traffic must stop

1 / 3

100%

Both instances in the pool. LB round-robins.

Read the walkthrough as text

Case, public REST API behind N identical pods.

Traffic: mostly short JSON GET/POST; a few long report downloads hold connections 30s. Single hostname api.example.com.

  1. Placement: Internet → managed L7 LB (or ingress controller) → pod pool in two AZs.
  2. Algorithm: Round-robin (or weighted) for JSON APIs; route /reports/* optionally to a pool with least-connections if downloads dominate connection count.
  3. Health: LB probes GET /health every 5s; 2 failures → remove from rotation; pod readiness gate ties to DB connectivity.
  4. Sessions: Stateless JWT; no cookie stickiness unless forced by legacy, if forced. Name uneven load and lost sessions on instance death.
  5. LB SPOF: Managed service spans AZs with its own control plane; you do not draw “one nginx on a laptop.”
  6. TLS: Terminate at LB or pass-through; mention certificate renewal on the LB.

Path-based L7. /v1/* to blue pool, /v2/* to green pool during migration, LB routes by prefix.

What a strong answer names. L7 vs L4 choice, algorithm tied to request shape, health + drain, stateless app, HA for the LB itself.

  1. Canary / blue-green: drain old pool via health check failure injection or weight to zero before kill.
  2. Cross-AZ latency: clients hit LB in their AZ; backends prefer same-AZ targets when possible.

Listen for: algorithm matched to connection duration, LB HA: health checks that reflect real readiness.

Napkin math

LB napkin

N = 3 API replicas (stateless)

algo = round-robin or least-conn

health = /healthz every 5s, remove on 2 fails

session = Redis / JWT, not process memory

per-instance capacity ≈ peak_QPS / N

LB spreads bytes. Shared state (or none) makes any replica safe.

3Answer it

How to answer in an interview

When the diagram has multiple app servers or “scale the API tier”:

  1. Front door, LB/ingress between clients and instances.
  2. L4 vs L7, path/host routing needs L7; pure TCP might be L4.
  3. Algorithm, round-robin vs least-conn with a workload reason.
  4. Health checks, what probe, what “unhealthy,” drain behavior.
  5. State, stateless + shared session, or stickiness and its cost.
  6. LB redundancy, managed HA, not one box.
  7. TLS and websockets, connection upgrade if relevant.

Tie every choice to request length, stickiness needs, and failure behavior.

Say: “L7 for path-based canary.” · “Least-conn for long downloads.” · “No stickiness if session in Redis.”

Answer checklist

1. What is being balanced (HTTP API, gRPC, TCP)? 2. Algorithm (round-robin, least-conn, consistent hash) and why. 3. Health checks and what happens on instance death. 4. Session stickiness, need it or make apps stateless? 5. LB itself as SPOF, managed LB / pair / Anycast.

Tradeoffs & numbers

  • Round-robin on long requests: one slow worker piles up connections, switch pool to least-connections.
  • Stickiness: simple for in-memory session; bad for balance and instance loss, prefer Redis session.
  • L7 inspection: flexible routing; slightly higher latency and CPU than L4.
  • DNS round-robin alone: no health awareness, always pair with LB layer.
  • Over-provisioning LB: LBs can saturate before apps; watch connection count and bandwidth.
  • Locality: same-AZ backends reduce cross-AZ latency; cross-zone LB for AZ failure.
  • When NOT L7: raw TCP game protocol, L4 pass-through, faster, less inspection.
  • Over-sticky: failover instance dies → users on that node error until reconnect.

Capacity: 50k concurrent connections on L7 may require dedicated NLB + gateway tier before REST CPU. Health: probe app readiness (DB up), not only process alive. - When NOT stickiness: stateless JWT + Redis session, round-robin stays fair and failover is painless. - gRPC/WebSocket: connection draining on deploy matters more than request count, mention preStop and LB deregistration delay.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Stateless HTTP round-robin

    Example. Ten identical API pods serving JSON CRUD. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.

    How. L7 LB round-robin; JWT/session in Redis; health checks remove bad pods. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.

  • Least-connections for long requests

    Example. Report exports hold connections 60s while lightweight APIs are 50ms. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.

    How. Use least-connections on the long-lived pool; keep round-robin for short APIs or separate path routes. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.

  • L7 routing / path-based

    Example. Canary: 5% traffic to `/v2` pool, rest to `/v1`. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.

    How. Route by path prefix or header; weight backends; mention rollback by draining canary. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.

  • Sticky sessions (and when to avoid)

    Example. Legacy app stores cart in local memory. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.

    How. Cookie-based affinity only if unavoidable; prefer shared session store so round-robin works. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

State behind the LB

Avoid. Session only in process memory. Next request hits another box and logs the user out.

Prefer. Stateless app + shared session store (or tokens). Any healthy instance can serve.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Stateful behind LB

    Why it happens. In-memory session on one instance breaks when the next request hits another server.

    How it fails. Shopping cart only in pod memory with round-robin and no stickiness.

    Fix. Shared session store (Redis) or JWT; or sticky sessions only with eyes open to imbalance.

  • No health checks

    Why it happens. Dead backends keep receiving traffic → errors and retry storms.

    How it fails. LB always forwards to all IPs even when half return 502.

    Fix. HTTP/TCP health checks; remove unhealthy targets; graceful drain on deploy.

  • LB as SPOF

    Why it happens. If the single LB dies, the whole service is down regardless of N app servers.

    How it fails. One self-managed nginx VM with no failover pair.

    Fix. Managed multi-AZ LB, HA pair with VIP, or cloud ingress with built-in redundancy.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons