1Understand
The idea
Notification fan-out delivers email, SMS, mobile push, and in-app alerts triggered by product events, without blocking the write path that created the event. The API should acknowledge quickly; channels send asynchronously with retries, provider rate limits, and user preferences (opt-out, quiet hours, channel choice).
Fan-out can be write-time (materialize one row per recipient when the event fires) or read-time (store the event; clients pull an inbox). Celebrity events punish naive write-time inserts. Good designs dedupe, batch, and queue provider calls.
Compliance: opt-in for marketing SMS, unsubscribe links on email, regional quiet hours, prefs are not nice-to-have at scale.
Intuition
Triggers and channels. Event: OrderShipped, MentionedInComment, TrialEnding. Channels: in-app inbox, mobile push (FCM/APNs), email, SMS. Each channel has different latency, cost, and user expectations, transactional SMS is not marketing email.
Async by default. Write path: persist business event → enqueue notification job → return 200/201. Never call SES/FCM synchronously in the critical user mutation unless regulatory instant delivery (rare).
Write-time vs read-time fan-out. Write-time: on celebrity post, insert N notification rows, simple reads, painful writes at scale. Read-time: store one activity event; followers page inbox at read, cheaper writes, heavier read aggregation. Hybrid: write-time for small fan-out; async worker batches for large.
Preferences. Per-user channel toggles, categories, quiet hours, locale. Check prefs before send; store dedupe keys (notify:mention:{commentId}:{userId}) so retries do not double-notify.
Providers. Email bounces, push token invalidation, SMS rate limits, handle partial failure, retry transient, DLQ permanent. Idempotent send records.
In-app inbox. Separate from push: durable rows users can mark read; badge counts; optional realtime via sockets for
Priority lanes: password reset > marketing digest, separate queues and worker pools so bulk camp does not starve transactional.
Template rendering: async worker builds email HTML; API stores intent only.
Unread math: badge may be denormalized counter or computed at read, state staleness. Bounce handling: hard bounce disables channel; soft bounce retries with cap. Celebrity path: paginate followers in worker; O(followers) never in request thread.
What you will be able to do
- Keep notification send off the user-facing write path
- Choose write-time vs read-time fan-out
- Respect preferences and provider rate limits
When this comes up
- Push notification
- Email/SMS
- Fan-out
- Preference center
- Quiet hours
2See it
Architecture sketch
Notify after the write
App commits, enqueue fan-out; workers talk to providers.
Walkthrough
Ship event → three channels
Prefs and idempotency matter
1 / 4
Order ships. API enqueues notification jobs (outbox). User request already returned.
Read the walkthrough as text
Case, user followed by 5M followers posts an update.
- Sync API: insert post row, return 201 immediately.
- Do not insert 5M inbox rows in the request.
- Enqueue
PostCreated {postId, authorId}to fan-out worker (or tiered: if follower count < 10k write-time else async). - Worker: paginate follower ids from index; batch enqueue per-1000 SendNotification tasks or bulk insert staged inbox rows in chunks.
- Push/email: separate rate-limited workers per provider; respect user prefs (followers who disabled push skip FCM).
- Dedupe: idempotency on
(eventId, userId, channel). - In-app: materialize inbox rows async; client polls or SSE/socket for unread count.
- Failure: provider 503 → retry with backoff; permanent bounce → disable email channel for user.
Transactional email: password reset, small fan-out, still async queue after token persisted; high priority queue.
- Rate limits: FCM/APNs/SES quotas, batch and throttle celebrity fan-out over minutes.
- Idempotency key:
(eventId, userId, channel)prevents duplicate push on retry.
Phrase: “API returns after persist; workers own SMTP/FCM.”
Napkin math
event = order.shipped
channels = push + email + in-app
enqueue per channel (idempotent)
provider outage → retry / DLQ
user prefs checked before send
Do not send email inside the request thread.
3Answer it
How to answer in an interview
When alerts, email, push, or “notify all followers” appears:
- Trigger event and which channels.
- Sync boundary, what user waits for.
- Fan-out strategy, write-time vs read-time vs hybrid; celebrity path.
- Preferences, dedupe, quiet hours.
- Queue + workers per channel; provider limits.
- Partial failure, which channels succeeded.
Never block HTTP on SMTP/FCM without explicit reason.
Say: “Hybrid fan-out for celebrity.” · “Prefs before send.” · “DLQ on provider hard bounce.”
Answer checklist
Tradeoffs & numbers
- Write-time inbox: fast reads; write spikes on viral events.
- Read-time feed: cheaper writes; complex “unread” at read.
- Multi-channel: better reach; more prefs and compliance (opt-in SMS).
- Sync send: simpler debugging; terrible tail latency on writes.
- Provider coupling: abstract adapters; still respect per-provider quotas.
- Realtime in-app: sockets add infra; polling inbox is OK for low urgency.
- When NOT multi-channel: user disabled all, skip send, still store in-app.
- Write-time 5M rows: minutes of lock time, always async tiered fan-out.
Provider math: SES 14 messages/s default, celebrity fan-out must batch over minutes. Dedupe key (eventId,userId,channel) on every worker. - Compliance: marketing SMS needs opt-in; transactional messages still respect quiet hours if product policy requires. - Partial success: API returns 201 after persist; per-channel failures logged, user may get push but not email; support uses send audit table.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Transactional email / SMS
Example. Password reset link and banking OTP must arrive within 60s even when login API is under load; user already sees “email sent” in UI.
How. Write token row + `NotificationIntent` in one txn; enqueue `HIGH` priority email worker; SES/Twilio async; dedupe `(userId, purpose, tokenId)`; HTTP returns after persist, not after SMTP 250 OK.
Mobile push fan-out
Example. Someone replies to your comment, user expects phone buzz within seconds while app may be backgrounded on iOS. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.
How. Event → queue → worker loads device tokens; FCM/APNs per token; drop invalid tokens; skip if push pref off; separate from in-app inbox row; retry 503 with backoff; never block comment POST on push latency.
In-app notification inbox
Example. Bell shows unread count and 30-day history; marketing must not pollute transactional thread. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.
How. Materialize `inbox` rows async or on read aggregation; `mark_read` API; optional WebSocket/SSE for badge only; category prefs filter rows before insert; SQL source for “what user already saw.”
Celebrity / large fan-out
Example. Creator with 10M followers publishes. Write path must stay sub-200ms while millions need push/email over minutes. API acks after persist; async channel workers with dedupe keys; respect opt-out/quiet hours and provider rate limits.
How. Post API inserts one row + enqueues `FanOutJob`; worker pages followers 5k/chunk; rate-limit SES/FCM; hybrid: small accounts write-time, large async; idempotency per `(postId, userId, channel)`.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Sync notify in request
Avoid. Await SendGrid inside POST /orders. Tail latency owns your p99.
Prefer. Commit + enqueue; workers and DLQ handle providers.
Weak sketch
Bad
Stronger sketch
Good
Sync send on write
Why it happens. Provider latency and failures inflate write-path tail latency and error rates.
How it fails. Create comment HTTP waits for SES and FCM before returning 201.
Fix. Persist event, enqueue per channel, return fast; workers send with retries.
No preferences
Why it happens. Users unsubscribe or churn when spammed on every channel regardless of settings.
How it fails. Every mention sends SMS, email, and push with no opt-out.
Fix. Preference center, quiet hours, category toggles; check before each channel send.
Naive fan-out
Why it happens. Synchronous N inserts or sends in the request does not scale for large follower graphs.
How it fails. Celebrity post inserts 5M notification rows inside the POST handler.
Fix. Async batched fan-out worker, hybrid read-time feed, rate-limited provider workers.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Transactional email path
On password reset, the API creates a single-use token in SQL and must email the user a link. The user expects the HTTP response in under 300ms. Where does SMTP run, and how do retries interact with duplicate requests?
- about 8 min
Mobile push fan-out
When someone likes your post, interested users with the app installed should get a push notification if they opted in. Sketch trigger → queue → FCM/APNs, including invalid token handling and preference checks.
- about 8 min
In-app inbox
Users need a notification bell with unread count and history, separate from mobile push. Compare materializing inbox rows at write-time vs aggregating at read-time for a user with 500 followers who receives a few mentions per day.
- about 9 min
Celebrity fan-out
A creator with 8M followers publishes a post. Product wants followers notified via push and in-app within minutes, not hours. The POST /posts API must stay fast. How do you fan out without inserting millions of rows synchronously?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.