1Understand
The idea
A load balancer spreads incoming connections and requests across a pool of healthy backends so no single instance carries the whole product. In interviews it is the glue between “we run N stateless APIs” and “clients actually hit one hostname.” The answer is not just “add an LB”, it is L4 vs L7, which algorithm fits the workload, health checks that stop sending traffic to dead nodes, and whether stickiness is required or a smell that the app is not stateless.
The LB itself can become a single point of failure, so managed load balancers, active/passive pairs, or Anycast matter. Good designs make session state live in a shared store or disappear entirely so round-robin stays fair.
Intuition
What is being balanced. HTTP/HTTPS APIs (L7), raw TCP (L4), gRPC streams, WebSockets. L4 LB is fast and connection-oriented; L7 can route by path, host header, cookies, or JWT claims.
Algorithms:
- Round-robin, fair for homogeneous short requests on stateless instances.
- Least connections, better when requests hold connections a long time (SSE, uploads, slow queries).
- Consistent hash, sticky by key at the LB for cacheable partitions; overlaps with the consistent-hashing pattern but at the edge.
- Weighted, canary or heterogeneous instance sizes.
Health checks. Active HTTP GET /health or TCP probe; unhealthy backends are drained, no new connections, in-flight requests may complete. Say graceful shutdown: app stops accepting, finishes work, then deregisters.
Statelessness vs stickiness. Ideal: any instance can serve any request; session in Redis/JWT. Sticky sessions (cookie to same backend) help legacy servers with in-memory session but hurt failover and uneven load, prefer shared session store.
LB HA. One hardware LB is a SPOF; use cloud managed LB, two LBs with floating IP, or Anycast anycast VIP.
What you will be able to do
- Place an LB in front of a replicated stateless tier
- Choose L4 vs L7 and a balancing algorithm with a reason
- Explain health checks and draining
When this comes up
- Stateless API tier
- Health checks
- L4 vs L7
- Sticky sessions
2See it
Architecture sketch
Stateless API behind a load balancer
LB spreads traffic; health checks remove bad instances. Make apps stateless or share session state.
Walkthrough
Health check drain
Instance dies, traffic must stop
1 / 3
Both instances in the pool. LB round-robins.
Read the walkthrough as text
Case, public REST API behind N identical pods.
Traffic: mostly short JSON GET/POST; a few long report downloads hold connections 30s. Single hostname api.example.com.
- Placement: Internet → managed L7 LB (or ingress controller) → pod pool in two AZs.
- Algorithm: Round-robin (or weighted) for JSON APIs; route
/reports/*optionally to a pool with least-connections if downloads dominate connection count. - Health: LB probes
GET /healthevery 5s; 2 failures → remove from rotation; pod readiness gate ties to DB connectivity. - Sessions: Stateless JWT; no cookie stickiness unless forced by legacy, if forced. Name uneven load and lost sessions on instance death.
- LB SPOF: Managed service spans AZs with its own control plane; you do not draw “one nginx on a laptop.”
- TLS: Terminate at LB or pass-through; mention certificate renewal on the LB.
Path-based L7. /v1/* to blue pool, /v2/* to green pool during migration, LB routes by prefix.
What a strong answer names. L7 vs L4 choice, algorithm tied to request shape, health + drain, stateless app, HA for the LB itself.
- Canary / blue-green: drain old pool via health check failure injection or weight to zero before kill.
- Cross-AZ latency: clients hit LB in their AZ; backends prefer same-AZ targets when possible.
Listen for: algorithm matched to connection duration, LB HA: health checks that reflect real readiness.
Napkin math
N = 3 API replicas (stateless)
algo = round-robin or least-conn
health = /healthz every 5s, remove on 2 fails
session = Redis / JWT, not process memory
per-instance capacity ≈ peak_QPS / N
LB spreads bytes. Shared state (or none) makes any replica safe.
3Answer it
How to answer in an interview
When the diagram has multiple app servers or “scale the API tier”:
- Front door, LB/ingress between clients and instances.
- L4 vs L7, path/host routing needs L7; pure TCP might be L4.
- Algorithm, round-robin vs least-conn with a workload reason.
- Health checks, what probe, what “unhealthy,” drain behavior.
- State, stateless + shared session, or stickiness and its cost.
- LB redundancy, managed HA, not one box.
- TLS and websockets, connection upgrade if relevant.
Tie every choice to request length, stickiness needs, and failure behavior.
Say: “L7 for path-based canary.” · “Least-conn for long downloads.” · “No stickiness if session in Redis.”
Answer checklist
Tradeoffs & numbers
- Round-robin on long requests: one slow worker piles up connections, switch pool to least-connections.
- Stickiness: simple for in-memory session; bad for balance and instance loss, prefer Redis session.
- L7 inspection: flexible routing; slightly higher latency and CPU than L4.
- DNS round-robin alone: no health awareness, always pair with LB layer.
- Over-provisioning LB: LBs can saturate before apps; watch connection count and bandwidth.
- Locality: same-AZ backends reduce cross-AZ latency; cross-zone LB for AZ failure.
- When NOT L7: raw TCP game protocol, L4 pass-through, faster, less inspection.
- Over-sticky: failover instance dies → users on that node error until reconnect.
Capacity: 50k concurrent connections on L7 may require dedicated NLB + gateway tier before REST CPU. Health: probe app readiness (DB up), not only process alive. - When NOT stickiness: stateless JWT + Redis session, round-robin stays fair and failover is painless. - gRPC/WebSocket: connection draining on deploy matters more than request count, mention preStop and LB deregistration delay.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Stateless HTTP round-robin
Example. Ten identical API pods serving JSON CRUD. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.
How. L7 LB round-robin; JWT/session in Redis; health checks remove bad pods. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.
Least-connections for long requests
Example. Report exports hold connections 60s while lightweight APIs are 50ms. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.
How. Use least-connections on the long-lived pool; keep round-robin for short APIs or separate path routes. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.
L7 routing / path-based
Example. Canary: 5% traffic to `/v2` pool, rest to `/v1`. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.
How. Route by path prefix or header; weight backends; mention rollback by draining canary. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.
Sticky sessions (and when to avoid)
Example. Legacy app stores cart in local memory. Pick L4 vs L7, health-check behavior, and whether stickiness is required; explain what happens when one AZ loses half the pool.
How. Cookie-based affinity only if unavoidable; prefer shared session store so round-robin works. Health checks drain bad nodes; round-robin only if app is stateless; call out LB as potential SPOF mitigation.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
State behind the LB
Avoid. Session only in process memory. Next request hits another box and logs the user out.
Prefer. Stateless app + shared session store (or tokens). Any healthy instance can serve.
Weak sketch
Bad
Stronger sketch
Good
Stateful behind LB
Why it happens. In-memory session on one instance breaks when the next request hits another server.
How it fails. Shopping cart only in pod memory with round-robin and no stickiness.
Fix. Shared session store (Redis) or JWT; or sticky sessions only with eyes open to imbalance.
No health checks
Why it happens. Dead backends keep receiving traffic → errors and retry storms.
How it fails. LB always forwards to all IPs even when half return 502.
Fix. HTTP/TCP health checks; remove unhealthy targets; graceful drain on deploy.
LB as SPOF
Why it happens. If the single LB dies, the whole service is down regardless of N app servers.
How it fails. One self-managed nginx VM with no failover pair.
Fix. Managed multi-AZ LB, HA pair with VIP, or cloud ingress with built-in redundancy.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 7 min
Stateless round-robin
You run 12 stateless API pods behind `api.example.com`. Sessions are in Redis; JWTs are validated on each request. What load balancer setup do you use (L4 vs L7), which balancing algorithm, and why skip sticky sessions? Mention health checks.
- about 8 min
Long-lived connections
Most API calls finish in 80ms, but `/export` opens a 2-minute streaming response. With one shared pool of 20 servers, why might round-robin hurt? What algorithm or routing split would you use?
- about 8 min
Path-based canary
You are rolling out `/v2` handlers on new pods while `/v1` stays on the stable pool. How does an L7 load balancer or ingress route traffic? How do you roll back if error rate spikes on v2?
- about 8 min
Sticky session tradeoff
A legacy service keeps user-specific cache only in process memory. The ops team wants cookie-based sticky sessions on the load balancer. Explain what problem stickiness solves, what breaks on instance failure, and what you would rather do for a greenfield design.
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.