Caching & Scale

Rate Limiting

Protect APIs and downstreams by limiting how fast a client can call you.

Intermediate, about 60 minutes to read, 4 open questions to practise.

1Understand

The idea

Rate limiting is a fuse in front of expensive or scarce work: APIs, login endpoints, downstream partners, or your own database. It answers *who* is limited (user, IP, tenant, API key), *how fast* they may go, and *what happens* when they exceed the limit; usually 429 with Retry-After.

The limiter must sit before the cost you are protecting, and in a fleet it must share state or you accidentally multiply quotas by instance count. Algorithm choice (fixed window, sliding window, token bucket) is about burst vs steady rate and fairness, not magic.

Say 429 is a contract: clients must backoff; partners integrate Retry-After. Limiters also protect your fleet from retry storms when downstream is slow.

Intuition

Why limit. Abuse (credential stuffing, scraping), fairness (free tier vs paid), and backpressure so one tenant cannot exhaust DB connections or partner API quotas.

Limit key must match identity you actually have: authenticated user id or API key; anonymous traffic → IP (know NAT/shared-IP false positives). Multi-tenant B2B → tenant id, not client IP behind corporate NAT.

Algorithms (interview level):

  • Fixed window, count requests per calendar minute; simple but allows 2× burst at window edges.
  • Sliding window, smoother; more state (logs or approximate counters).
  • Token bucket, refill tokens at steady rate; allows controlled bursts up to bucket size (good for “100/sec steady, burst 20”).
  • Leaky bucket, smooths output rate; shapes traffic to downstream.

Placement. Edge API gateway or first middleware before auth-heavy DB work. Rate limit after you have already run the expensive query is “too late.”

Distributed fleets. Counters in Redis (or similar) with atomic increment; local in-memory counters are only OK if the limit is per-instance *by design*. Health checks and internal traffic often bypass limits.

Client contract. 429 Too Many Requests, Retry-After header, optional body with

Global vs local budget: “1000 req/min per API key” is product intent; per-pod counters violate that unless you multiply quota by replicas accidentally.

Adaptive limits: shed load when DB pool > 80%, combine with queue depth, not only request count.quota reset time.

What you will be able to do

  • Choose a limit key and window that match the threat
  • Pick an algorithm and explain burst vs steady rate
  • Place the limiter so it actually protects the expensive resource

When this comes up

  • Abuse
  • Fairness
  • 429 Too Many Requests
  • Per-user or per-IP quotas

2See it

Architecture sketch

Limiter in front of expensive work

Gateway checks a shared counter first. Rejected calls never hit the app or DB.

100%
ClientGatewayCacheServiceStore

Walkthrough

Allow vs 429

Token bucket on the gateway

1 / 4

100%

Request hits the gateway. Limit key is userId over a 60s window.

Read the walkthrough as text

Case, public REST API with free and paid tiers.

Threat: one API key scrapes the catalog; another shared office IP hits login with many accounts.

  1. Authenticated catalog reads, limit key = api_key:{id}; token bucket 100 req/s steady, burst 300; enforced at API gateway after API key validation (cheap) and before DB.
  2. Login, limit key = ip:{clientIp} + separate account:{username} lockout; low fixed window (e.g. 10/min/IP) because unauthenticated.
  3. Implementation, Redis INCR with TTL for sliding-ish windows, or dedicated limiter library; all app instances share Redis so 10 pods do not each multiply quota.
  4. Response, 429 with Retry-After; paid tier higher limits stored in key metadata.
  5. Bypass, synthetic monitoring uses separate keys with high ceiling.

What a strong answer names. Keys, algorithm (token bucket for burst), gateway placement, shared store, 429 behavior, and why IP limit on login is wrong for catalog reads

  1. Observability: metric rate_limit_exceeded_total{route,key_type}, distinguish abuse from misconfigured partner.
  1. Grace: soft limit (delay) vs hard 429, interviews usually want hard reject before expensive work.behind one NAT.

Napkin math

Quota napkin

key = userId (or apiKey)

limit = 100 req / 60s

algo = token bucket (burst 20)

store = Redis INCR + EXPIRE / Lua

deny → 429 + Retry-After

Limit before the expensive path. Shared store so N replicas do not multiply quota.

3Answer it

How to answer in an interview

When the prompt mentions abuse, quotas, or protecting downstreams:

  1. Resource protected, DB, partner API, email send, expensive ML call.
  2. Identity / key, user, API key, IP, tenant; justify for auth model.
  3. Limit shape, requests per second/minute/day; burst allowance.
  4. Algorithm, tie to burst vs steady (token bucket is the usual hero).
  5. Where it runs, gateway vs service middleware; before expensive work.
  6. Distributed, shared Redis vs per-node mistake.
  7. Client UX, 429, Retry-After, differentiation of tiers.

One sentence on what you would *not* limit (health, static assets)

Say: “Enforce at gateway after cheap auth, before DB.” · “Token bucket for burst + steady.” · “Shared Redis counters for fleet-wide quota.”shows maturity.

Answer checklist

1. What is limited (user, IP, API key, tenant) and over what window? 2. Algorithm (fixed window, sliding window, token bucket, leaky bucket) and why. 3. Where it runs (gateway, middleware, Redis). 4. Response to the client (429, Retry-After) and bypass for health checks. 5. Distributed limiters: shared store vs local approximate.

Tradeoffs & numbers

  • Strict global limits: fair but need shared state; Redis becomes a dependency, cache/local fallback risks over-admission.
  • Approximate local limits: survive Redis blip but effective quota scales with replica count, state the risk.
  • IP limits: cheap for login abuse; unfair behind carrier-grade NAT.
  • Sliding vs fixed window: sliding is fairer; fixed is simpler at very high QPS.
  • Reject vs queue: rate limits usually reject fast (429); queuing is a different pattern (backpressure queue).
  • False negatives: whitelisting partners; cost of bypass holes, monitor anomalously high 429 rates.
  • When NOT to rate-limit: internal mesh with mTLS and fixed callers, use quotas at integration boundary instead.
  • Fixed window at 100k QPS: edge doubling at minute boundary, prefer sliding or token bucket for public API.
  • Cost of Redis limiter: extra RTT per request, acceptable at gateway, not inside inner loop per row.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Per-user API quota

    Example. Paid API customers get higher limits than free tier. Specify limit key (user, API key, IP, tenant), steady vs burst allowance, HTTP 429 + Retry-After, and where enforcement sits before the DB or partner call.

    How. Key = API key or user id; token bucket per tier; 429 + Retry-After; enforce at gateway before DB. Enforce at gateway; return 429 early; distributed store if fleet >1; never rate-limit after the expensive query.

  • IP-based abuse shield

    Example. Unauthenticated login endpoint hit by credential stuffing. Specify limit key (user, API key, IP, tenant), steady vs burst allowance, HTTP 429 + Retry-After, and where enforcement sits before the DB or partner call.

    How. Key = client IP (and optionally username); low fixed window; complement with CAPTCHA/lockout, not for NAT-heavy catalog.

  • Token bucket bursts

    Example. Clients poll status every second but may burst 50 at job start. Specify limit key (user, API key, IP, tenant), steady vs burst allowance, HTTP 429 + Retry-After, and where enforcement sits before the DB or partner call.

    How. Steady refill rate + bucket size; explain burst allowance vs fixed window edge doubling. Enforce at gateway; return 429 early; distributed store if fleet >1; never rate-limit after the expensive query.

  • Distributed limiter with Redis

    Example. Fleet of 20 API servers must share one global quota. Specify limit key (user, API key, IP, tenant), steady vs burst allowance, HTTP 429 + Retry-After, and where enforcement sits before the DB or partner call.

    How. Central counter in Redis with atomic ops; never per-instance memory-only for global limits. Enforce at gateway; return 429 early; distributed store if fleet >1; never rate-limit after the expensive query.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Where the counter lives

Avoid. Per-instance memory counter. 10 replicas ≈ 10× the quota.

Prefer. Shared Redis (or edge) limiter. Fleet size does not multiply allowance.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Wrong limit key

    Why it happens. The wrong identity dimension blocks innocents or misses abusers.

    How it fails. Per-IP limit on a mobile app API where thousands of users share one carrier NAT.

    Fix. Match key to auth: API key/user id when authenticated; IP for login abuse; tenant id for B2B.

  • Too late

    Why it happens. Limiting after the expensive query already ran does not protect the database.

    How it fails. Rate limit checked inside the handler after a heavy join and external API call.

    Fix. Enforce at gateway or first middleware before DB/partner calls; reject with 429 early.

  • Local-only in a fleet

    Why it happens. Per-process counters multiply the effective limit by instance count.

    How it fails. 100 req/s in-memory per pod × 20 pods ≈ 2000 req/s effective for one API key.

    Fix. Shared store (Redis) or explicit per-instance limits only if product intent is per-node.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons