1Understand
The idea
Schedulers and workflows run work that must happen later, repeatedly, or across multiple steps without tying up an HTTP thread. Trials ending, invoice cron, approval chains, and refund pipelines need durable timers, retries, and idempotency under at-least-once execution.
A cron file on one laptop pod is not HA; await sleep(14 days) in Node dies on deploy. Interviews compare managed schedulers, delay queues, DB `run_at` polling, and workflow engines (Temporal, Step Functions) based on human-in-the-loop complexity.
Every step must tolerate duplicate delivery: cron overlap during failover, visibility timeout redelivery, or at-least-once queue semantics. Observability for missed crons is part of the design, silent failure until month-end is a classic production horror story.
Intuition
Job categories. One-shot delay: send reminder T+24h. Recurring: 0 2 * * * rollup. Workflow: state machine with human gates (AWAITING_MANAGER).
Timer storage. In-process timers lost on restart. Delay queue: SQS DelaySeconds / scheduled visibility; scheduler: Cloud Scheduler / k8s CronJob firing HTTP or enqueue; DB: scheduled_jobs(run_at, payload) polled by workers with FOR UPDATE SKIP LOCKED; workflow engine: durable timers in service history.
Idempotency. Keys like trial-end:{userId}:{period}; check status ALREADY_CHARGED before side effect; ledger upsert.
Visibility timeout. Worker dies mid-job → message visible again → second worker continues if steps checkpointed.
Workflow versioning. Changing step code mid-flight, engines replay from history; DIY cron needs explicit version on row.
Time zones. Bill on tenant TZ or UTC explicitly; DST jumps break naive local(2am), store UTC run_at.
Human steps. Timeouts escalate; compensate on reject (saga link).
Poison jobs. After max retries, DLQ with payload preserved; admin replay UI requeues with same idempotency key, workflows and queues share failure vocabulary.
Backpressure. If downstream billing API slows, delay queue depth rises, scale workers or throttle scheduler enqueue rate so timers do not pile unbounded in memory.
Human approvals. Manager step may wait days, workflow engine persists state; HTTP request already returned; user polls status URL.
Priority queues. Trial-end charge beats marketing email, separate queues so bulk campaigns never starve billing timers.
What you will be able to do
- Pick cron vs queue delay vs workflow engine
- Make job steps idempotent under retry
- Observe lag and failed runs
When this comes up
- Cron
- Delayed job
- Saga workflow
- Exactly-once job
2See it
Architecture sketch
Durable workflow engine
Long-running steps with state, retries, and human tasks. Not a brittle cron chain.
Walkthrough
KYC multi-step
Survive process restarts
1 / 4
Workflow starts and checkpoints “collect docs” in durable state.
Read the walkthrough as text
Worked example, SaaS trial ends day 14: charge or downgrade.
- Signup: insert
trial_ends_atUTC; enqueueTrialEnd {userId}scheduled at that instant (Scheduler → SQS or Temporal timer). - Never
setTimeoutin API. - Worker fires: load user; if
cancelled→ exit 0; ifTRIAL→ call billing with idempotency keytrial-end:{userId}. - Duplicate fire: second run sees
status=ACTIVE_PAID→ no-op. - Payment 503: exponential backoff; after N tries → DLQ + pager; user stays
TRIALwith banner, honest UX. - Observability: metric
trial_end_jobs_processed; alert if no success in 24h window. - Multi-step upgrade: optional workflow
TRIAL → PAYMENT_PENDING → ACTIVEwith webhook from Stripe as signal.
Nightly report variant: CronJob 02:00 UTC; idempotent report_runs row per date; alert if last_success stale > 26h. Refund workflow variant: three human steps with 72h SLA timer per step; escalate to finance queue on timeout.
What interviewers listen for: durable scheduler; idempotency; not in-request sleep; missed-run alerts; timezone clarity.
Napkin math
state machine with durable checkpoints
each step: enqueue → worker → record result
retry with backoff; max attempts then fail
human task = wait state, not busy poll
timers / SLA alerts on stuck runs
Long-running work lives in an engine, not a fragile cron chain.
3Answer it
How to answer in an interview
When delayed billing, cron, approvals, reminders:
- Job type, delay / recurring / multi-step.
- Infrastructure, scheduler, queue, workflow engine. Pick one with reason.
- Idempotency, concrete keys and status checks.
- Failure, retry, DLQ, human escalation.
- Observability, last success timestamp, lag.
- Clocks, UTC storage, tenant display rules.
Phrases: “durable timer,” “at-least-once safe handler,” “SKIP LOCKED job polling,” “alert on missed cron window.”
- Compare options, Scheduler vs Temporal in one sentence tied to human approval steps.
- Escalation. Name on-call owner when job stuck past SLA; finance cares about missed billing more than pod restarts.
Answer checklist
Tradeoffs & numbers
- k8s CronJob: simple; missed if cluster unhealthy, external heartbeat monitor.
- Delay queue: durable; max delay limits on some brokers; not always exact to the millisecond.
- Temporal/Step Functions: best long workflows + humans; learning curve and cost.
- DB polling: easy prototype; index
run_atand cap batch size at scale. - Exactly-once jobs: rare; design at-least-once + idempotency.
- When NOT workflow engine: single cron SQL report, Scheduler + idempotent script enough.
- Clock skew: NTP matters; use external time source for billing cutoffs; never
Date.now()alone across pods for legal deadlines. - Interview tip: contrast in-process timer vs queue timer with deploy-crash story, one sentence, high impact.
- Runbook: name owner for missed cron and max acceptable staleness of derived tables, finance cares about this more than k8s.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Recurring cron
Example. Aggregate usage metrics every night for billing; missed run loses revenue. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.
How. UTC CronJob + idempotent insert per `billing_date`; PagerDuty if `last_success` > 26h stale. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.
Delayed single job
Example. Abandoned cart email 2 hours after last activity. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.
How. Schedule message with `deliverAt`; cancel if purchase event arrives first; idempotent send per cart id. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.
Multi-step workflow
Example. Vendor payout: invoice submitted → finance approve → ACH sent. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.
How. Persist `workflow_state`; each transition event handler idempotent; timeout escalates to manual queue. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.
At-least-once job runner
Example. Worker killed mid-export after writing half the rows to S3. Durable timers/queues; every step idempotent under at-least-once; visibility into missed crons and poison messages.
How. Checkpoint part numbers; redelivery resumes or overwrites idempotent S3 key `export/{id}/part-NN`. Delay queue or `run_at` table; heartbeat lease; DLQ for poison; metrics on backlog age.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Cron spaghetti
Avoid. Five crons polling tables with no single state machine. Partial failures orphan rows.
Prefer. One durable workflow with explicit states, retries, and timeouts.
Weak sketch
Bad
Stronger sketch
Good
Sleep in request
Why it happens. HTTP workers scale horizontally; in-memory sleep is lost on crash/deploy and holds precious connections.
How it fails. Signup handler awaits seven days then charges card inside the same Lambda invocation.
Fix. Persist schedule; delay queue or scheduler fires durable worker later.
Non-idempotent job
Why it happens. Cron overlap and message redelivery double-charge, double-email, or duplicate ledger entries.
How it fails. Billing job always POSTs charge without checking prior `invoice_id` or provider idempotency key.
Fix. Status table, idempotency keys, upsert ledger rows safe on retry.
Silent cron failure
Why it happens. Batch jobs fail quietly until downstream business processes discover missing data weeks later.
How it fails. Nightly aggregation CronJob logs error to stdout only; no metric on last success.
Fix. Heartbeat metric, alert on stale `last_success`, DLQ for failed job payloads.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Recurring cron job
A nightly job aggregates usage into billing tables at 02:00 UTC. It must not double-run on failover, and on-call must know if it skipped a night. How do you schedule, execute, and observe it?
- about 8 min
Delayed reminder job
Users schedule “remind me tomorrow at 9am” for a task. They may complete the task early. Design durable scheduling, cancellation, and duplicate-fire safety.
- about 9 min
Multi-step refund workflow
Refunds require user request, manager approval within 48h, then payment-provider call. Outline persisted states, idempotent transitions, and what happens if the provider times out mid-flight.
- about 8 min
At-least-once job runner
A queue worker generates PDF invoices. It sometimes crashes after uploading to S3 but before ACKing the message, so the job runs twice. How do you prevent duplicate invoices?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.