Messaging & Async

Notifications Fan-out

Deliver email, push, SMS, or in-app alerts reliably without blocking the write path.

Intermediate, about 65 minutes to read, 4 open questions to practise.

1Understand

The idea

Notification fan-out delivers email, SMS, mobile push, and in-app alerts triggered by product events, without blocking the write path that created the event. The API should acknowledge quickly; channels send asynchronously with retries, provider rate limits, and user preferences (opt-out, quiet hours, channel choice).

Fan-out can be write-time (materialize one row per recipient when the event fires) or read-time (store the event; clients pull an inbox). Celebrity events punish naive write-time inserts. Good designs dedupe, batch, and queue provider calls.

Compliance: opt-in for marketing SMS, unsubscribe links on email, regional quiet hours, prefs are not nice-to-have at scale.

Intuition

Triggers and channels. Event: OrderShipped, MentionedInComment, TrialEnding. Channels: in-app inbox, mobile push (FCM/APNs), email, SMS. Each channel has different latency, cost, and user expectations, transactional SMS is not marketing email.

Async by default. Write path: persist business event → enqueue notification job → return 200/201. Never call SES/FCM synchronously in the critical user mutation unless regulatory instant delivery (rare).

Write-time vs read-time fan-out. Write-time: on celebrity post, insert N notification rows, simple reads, painful writes at scale. Read-time: store one activity event; followers page inbox at read, cheaper writes, heavier read aggregation. Hybrid: write-time for small fan-out; async worker batches for large.

Preferences. Per-user channel toggles, categories, quiet hours, locale. Check prefs before send; store dedupe keys (notify:mention:{commentId}:{userId}) so retries do not double-notify.

Providers. Email bounces, push token invalidation, SMS rate limits, handle partial failure, retry transient, DLQ permanent. Idempotent send records.

In-app inbox. Separate from push: durable rows users can mark read; badge counts; optional realtime via sockets for

Priority lanes: password reset > marketing digest, separate queues and worker pools so bulk camp does not starve transactional.

Template rendering: async worker builds email HTML; API stores intent only.

Unread math: badge may be denormalized counter or computed at read, state staleness. Bounce handling: hard bounce disables channel; soft bounce retries with cap. Celebrity path: paginate followers in worker; O(followers) never in request thread.

What you will be able to do

  • Keep notification send off the user-facing write path
  • Choose write-time vs read-time fan-out
  • Respect preferences and provider rate limits

When this comes up

  • Push notification
  • Email/SMS
  • Fan-out
  • Preference center
  • Quiet hours

2See it

Architecture sketch

Notify after the write

App commits, enqueue fan-out; workers talk to providers.

100%
ClientServiceStoreQueueGateway

Walkthrough

Ship event → three channels

Prefs and idempotency matter

1 / 4

100%

Order ships. API enqueues notification jobs (outbox). User request already returned.

Read the walkthrough as text

Case, user followed by 5M followers posts an update.

  1. Sync API: insert post row, return 201 immediately.
  2. Do not insert 5M inbox rows in the request.
  3. Enqueue PostCreated {postId, authorId} to fan-out worker (or tiered: if follower count < 10k write-time else async).
  4. Worker: paginate follower ids from index; batch enqueue per-1000 SendNotification tasks or bulk insert staged inbox rows in chunks.
  5. Push/email: separate rate-limited workers per provider; respect user prefs (followers who disabled push skip FCM).
  6. Dedupe: idempotency on (eventId, userId, channel).
  7. In-app: materialize inbox rows async; client polls or SSE/socket for unread count.
  8. Failure: provider 503 → retry with backoff; permanent bounce → disable email channel for user.

Transactional email: password reset, small fan-out, still async queue after token persisted; high priority queue.

  1. Rate limits: FCM/APNs/SES quotas, batch and throttle celebrity fan-out over minutes.
  2. Idempotency key: (eventId, userId, channel) prevents duplicate push on retry.

Phrase: “API returns after persist; workers own SMTP/FCM.”

Napkin math

Fan-out napkin

event = order.shipped

channels = push + email + in-app

enqueue per channel (idempotent)

provider outage → retry / DLQ

user prefs checked before send

Do not send email inside the request thread.

3Answer it

How to answer in an interview

When alerts, email, push, or “notify all followers” appears:

  1. Trigger event and which channels.
  2. Sync boundary, what user waits for.
  3. Fan-out strategy, write-time vs read-time vs hybrid; celebrity path.
  4. Preferences, dedupe, quiet hours.
  5. Queue + workers per channel; provider limits.
  6. Partial failure, which channels succeeded.

Never block HTTP on SMTP/FCM without explicit reason.

Say: “Hybrid fan-out for celebrity.” · “Prefs before send.” · “DLQ on provider hard bounce.”

Answer checklist

1. Trigger event and channels (in-app, push, email, SMS). 2. Sync ack vs async send. 3. Fan-out model (write-time vs read-time / feed). 4. Preferences, dedupe, retries, provider limits. 5. Failure: provider down, partial delivery.

Tradeoffs & numbers

  • Write-time inbox: fast reads; write spikes on viral events.
  • Read-time feed: cheaper writes; complex “unread” at read.
  • Multi-channel: better reach; more prefs and compliance (opt-in SMS).
  • Sync send: simpler debugging; terrible tail latency on writes.
  • Provider coupling: abstract adapters; still respect per-provider quotas.
  • Realtime in-app: sockets add infra; polling inbox is OK for low urgency.
  • When NOT multi-channel: user disabled all, skip send, still store in-app.
  • Write-time 5M rows: minutes of lock time, always async tiered fan-out.

Provider math: SES 14 messages/s default, celebrity fan-out must batch over minutes. Dedupe key (eventId,userId,channel) on every worker. - Compliance: marketing SMS needs opt-in; transactional messages still respect quiet hours if product policy requires. - Partial success: API returns 201 after persist; per-channel failures logged, user may get push but not email; support uses send audit table.

Kinds of questions (4)

Same pattern, different interview surfaces. Recognise which variant you are in before you answer.

  • Transactional email / SMS

    Example. Password reset link and banking OTP must arrive within 60s even when login API is under load; user already sees “email sent” in UI.

    How. Write token row + `NotificationIntent` in one txn; enqueue `HIGH` priority email worker; SES/Twilio async; dedupe `(userId, purpose, tokenId)`; HTTP returns after persist, not after SMTP 250 OK.

  • Mobile push fan-out

    Example. Someone replies to your comment, user expects phone buzz within seconds while app may be backgrounded on iOS. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.

    How. Event → queue → worker loads device tokens; FCM/APNs per token; drop invalid tokens; skip if push pref off; separate from in-app inbox row; retry 503 with backoff; never block comment POST on push latency.

  • In-app notification inbox

    Example. Bell shows unread count and 30-day history; marketing must not pollute transactional thread. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.

    How. Materialize `inbox` rows async or on read aggregation; `mark_read` API; optional WebSocket/SSE for badge only; category prefs filter rows before insert; SQL source for “what user already saw.”

  • Celebrity / large fan-out

    Example. Creator with 10M followers publishes. Write path must stay sub-200ms while millions need push/email over minutes. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.

    How. Post API inserts one row + enqueues `FanOutJob`; worker pages followers 5k/chunk; rate-limit SES/FCM; hybrid: small accounts write-time, large async; idempotency per `(postId, userId, channel)`.

Common mistakes

Failure modes the AI coach watches for when you practise open questions.

Sync notify in request

Avoid. Await SendGrid inside POST /orders. Tail latency owns your p99.

Prefer. Commit + enqueue; workers and DLQ handle providers.

Weak sketch

Bad

100%

Stronger sketch

Good

100%
  • Sync send on write

    Why it happens. Provider latency and failures inflate write-path tail latency and error rates.

    How it fails. Create comment HTTP waits for SES and FCM before returning 201.

    Fix. Persist event, enqueue per channel, return fast; workers send with retries.

  • No preferences

    Why it happens. Users unsubscribe or churn when spammed on every channel regardless of settings.

    How it fails. Every mention sends SMS, email, and push with no opt-out.

    Fix. Preference center, quiet hours, category toggles; check before each channel send.

  • Naive fan-out

    Why it happens. Synchronous N inserts or sends in the request does not scale for large follower graphs.

    How it fails. Celebrity post inserts 5M notification rows inside the POST handler.

    Fix. Async batched fan-out worker, hybrid read-time feed, rate-limited provider workers.

4Practise

Concept check

A short quiz on this lesson before open questions. Optional, and it does not gate practice.

Open questions

Ready to practise?

Write an answer, get AI coaching, then revise. You are not building a real system.

Practise with an open question

All systems lessons