1Understand
The idea
Messaging moves slow, bursty, or fan-out work off the user’s request path: emails, thumbnails, search index updates, billing webhooks. Producers publish; queues or topics buffer; consumers process with retries and a dead-letter queue (DLQ) when poison messages appear.
Brokers usually give at-least-once delivery, so consumers must be idempotent, the same message twice must not double-charge or spam. Backpressure (lag, shed load, scale consumers) is part of the design, not an afterthought.
Name ordering scope (per-partition), payload size limit (reference in S3), and poison message path to DLQ before ops pages at 3am.
Intuition
Queue vs pub/sub. Queue (point-to-point): one consumer group processes each message once (competing consumers). Pub/sub (topic): each subscriber gets a copy, fan-out notifications, multiple indexes.
Why async. User should not wait for email, PDF render, fraud scoring on 3rd party, or reindexing 50 search fields. API acknowledges fast; worker does the rest.
Delivery semantics. Many brokers guarantee at-least-once (ack after processing; redelivery on crash). Exactly-once end-to-end is hard, design idempotent consumers (dedupe by message id, upsert, outbox).
Retries & DLQ. Transient failures → backoff retry with max attempts; poison message → dead-letter queue for inspection. Ordering: often per-partition or per-key ordering, not global.
Backpressure. Consumer lag metrics; scale consumers; slow down producers; drop/shed only if product allows. Broker disk full is an outage, monitor retention.
Broker failure. Multi-AZ cluster, replicated logs; consumers pause and resume; outbox pattern keeps DB and publish consistent.
Visibility timeout / ack: process message, then ack; crash before ack → redelivery, design consumers idempotent. Priority queues for payment email vs marketing drip.
Schema evolution: version envelope (eventType, schemaVersion, payload) so workers tolerate rolling deploys. Observability: chart consumer lag, processing time, and DLQ insert rate per queue, interviewers tie lag SLO to product promise (“receipt email within 60s”).
What you will be able to do
- Choose async when the user should not wait on slow work
- Design consumers for at-least-once with idempotency
- Plan retries and dead letters
When this comes up
- Async processing
- Decouple producers
- Retry / DLQ
- At-least-once delivery
2See it
Architecture sketch
Async after write
User request commits quickly; slow work runs on a queue with retries / DLQ.
Walkthrough
Decouple the slow path
Email / thumbnail / fan-out after respond
1 / 4
User POSTs. API validates and starts a transaction.
Read the walkthrough as text
Case, user uploads video; API must return quickly.
Post-upload: transcode, generate thumbnails, update search index, notify followers.
- Sync path: API writes metadata to SQL, puts object in S3, enqueues
VideoUploaded {videoId}to queue, returns201with video id. - Workers: transcode worker, thumbnail worker (competing consumers on same queue or separate queues by SLA).
- Pub/sub side:
VideoPublishedtopic → email service + push notification service each subscribe. - Idempotency: worker keys work by
videoId; reruns skip if status alreadyTRANSCODED. - Retries: 3 attempts exponential backoff; then DLQ + alert.
- Ordering: per
videoIdpartition if steps must be serial (transcode before notify). - Lag: scale consumers when queue depth > threshold; do not block upload API on transcode.
Outbox optional: write event row in same SQL txn as insert; relay publishes, avoids “saved to DB but never enqueued.”
- SLO on lag: “p99 enqueue-to-process under 30s”, scale consumers on depth metric.
- Ordering: same
orderIdpartition key so fulfillment steps stay serial.
Listen for: fast API ack, idempotent worker, DLQ, not doing 30s SMTP in HTTP.
Napkin math
write path: commit DB → enqueue → 200
delivery = at-least-once (retry + ack)
consumer must be idempotent (dedupe key)
visibility timeout ≈ max processing + buffer
fail N times → DLQ, alert
Outbox/CDC if you cannot lose the event after commit.
3Answer it
How to answer in an interview
When slow work, notifications, or decoupling appears:
- What leaves sync path and user-visible deadline.
- Queue vs topic, one worker vs many subscribers.
- Message shape, ids, idempotency key, trace id.
- Semantics, at-least-once + idempotent handler.
- Retries, DLQ, poison pills.
- Ordering, per-key if needed.
- Backpressure & broker HA.
Never put 30s of fan-out inside the HTTP handler without saying why.
Say: “At-least-once + idempotent consumer.” · “DLQ after N tries.” · “Outbox if DB and broker must agree.”
Answer checklist
Tradeoffs & numbers
- Sync simplicity vs async complexity: ops, monitoring, duplicate delivery.
- Strong ordering: fewer parallel consumers per partition.
- Big messages: store blob in object store; queue carries reference only.
- In-process async: no cross-host durability, fine for dev, not for payments email.
- Kafka vs SQS/Rabbit: log retention, consumer groups, ops cost. Pick one with reason.
- Transactional outbox: stronger consistency; extra table and relay worker.
- When NOT messaging: synchronous read path needing immediate consistency, queue adds lag.
- Kafka log retention: disk cost vs replay benefit, set retention to business need.
Interview numbers: if API acks in 50ms but email p99 is 8s, users still win, quote both. Poison pill after 5 retries → DLQ with alert; lag > 5min → scale consumers or throttle producers. - When NOT a queue: user must see synchronous result (balance after transfer), say you still use DB transaction, not Kafka, for that read.
Ordering example: partition key = orderId so fulfillment steps run serial; global order across users is unnecessary and expensive. Payload rule: queue carries ids and metadata; multi-megabyte blobs live in object storage with pointer in message body.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Task queue after write
Example. Checkout API commits order in Postgres; customer gets 201 in ~80ms while fulfillment, email receipt, and warehouse pick-list run in workers.
How. Same transaction writes order row + outbox `OrderCreated`; relay publishes after commit; competing consumers on `fulfillment` queue; handler idempotent on `orderId`; 3 retries then DLQ with pager.
Pub/sub fan-out
Example. New product listed, search index, analytics pipeline, and partner webhook must all react without one serial worker bottleneck.
How. Publish `ProductListed` to topic; three independent subscriber groups each ack after idempotent processing; poison messages route to per-subscriber DLQ; no shared queue that couples search lag to webhook SLA.
Delayed / scheduled jobs
Example. SaaS trial ends in 14 days, charge card or downgrade plan at exact UTC instant without `sleep` in the HTTP handler. State delivery semantics (at-least-once), idempotency key, DLQ path, and how consumer lag pages before user-visible delays.
How. Scheduler or delay queue with visibility timeout fires `TrialEnd {userId}`; worker calls billing with idempotency key; duplicate delivery checks `subscription_status` before charge; alert if job queue stalled.
DLQ and retries
Example. Thumbnail worker throws on corrupt image bytes; message redelivers forever and blocks partition progress. State delivery semantics (at-least-once), idempotency key, DLQ path, and how consumer lag pages before user-visible delays.
How. Exponential backoff max 5 attempts; move to DLQ with payload + stack trace; dashboard DLQ depth; replay tool after fix deploy; never infinite ack/nack loop without ops signal.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Lose the event
Avoid. Commit DB, then publish. Crash between them and the event is gone.
Prefer. Outbox (or transaction log CDC): event durability matches the write.
Weak sketch
Bad
Stronger sketch
Good
Keeps everything sync
Why it happens. Synchronous side effects on the request path inflate p99 latency and tie user-visible success to SMTP, transcode, or third-party SLAs you do not control. One slow partner becomes your API outage.
How it fails. POST /orders commits SQL, then the handler calls SES, Stripe capture, and inventory RPC sequentially before returning 201, mobile clients timeout and retry into duplicate risk.
Fix. Commit durable state + enqueue domain events; return 201 with resource id immediately; workers own slow IO with idempotent handlers, retries, and DLQ for poison messages.
No idempotent consumer
Why it happens. Brokers redeliver on crash, visibility timeout, or network blip, without idempotent consumers, “exactly-once” intent becomes duplicate money movement, duplicate emails, or double inventory decrement.
How it fails. Payment worker always `INSERT INTO charges` on every message with no natural key; the same `paymentId` message processed twice creates two charge rows.
Fix. Upsert on business id, status guard (`if status < PAID`), or dedupe table keyed by `messageId`; design so second delivery is a no-op with same side effect as first.
Ignores backpressure
Why it happens. Unbounded enqueue rate with fixed consumers turns queues into hours-long lag or fills broker disk, users see “accepted” events that never process, and one bad deploy can require mass replay.
How it fails. Flash sale publishes 2M events with no consumer autoscaling, no `queue_depth` alert, and 7-day retention on a small Kafka cluster, disk full stops all consumers.
Fix. Metric lag/depth; scale consumers on threshold; optional producer throttle; DLQ policy; retention sized to replay window; page when age-of-oldest-message exceeds product SLA.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Queue after write
On signup, you must create the user in SQL, send a welcome email, and enqueue default settings setup. The user should see “account created” in under 300ms. What happens synchronously in the API vs asynchronously in workers?
- about 8 min
Pub/sub fan-out
When a post is published, search index, analytics, and mobile push must all update. Queue with one consumer is too slow serially. Compare a task queue vs a pub/sub topic. Sketch message flow and subscriber responsibilities.
- about 8 min
Delayed jobs
Users start a 7-day free trial; on day 7 you charge or send a reminder. You cannot keep a thread sleeping for 7 days. How do you schedule the job, what idempotency do you need if the scheduler fires twice, and what happens if the user cancels on day 3?
- about 8 min
DLQ and retries
A payment webhook consumer throws on a new partner payload shape. Messages retry forever and block the queue. Design retry policy, dead-letter handling, and how ops replays after a fix ships.
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.