1Understand
The idea
Stream and batch processing move product events into shapes humans and dashboards can use: aggregates, warehouse tables, search documents, fraud scores. The interview fork is freshness: if stakeholders need “yesterday’s revenue” by 6am, a partitioned batch job beats a Kafka cluster; if fraud must fire within seconds of a card swipe, you need consumers, windows, and partition-scaled parallelism.
Streams assume unbounded input, at-least-once delivery, and order only within a partition key. Batch assumes bounded input (a day’s parquet files), high throughput, and hours of latency tolerance. CDC (change data capture) bridges OLTP to the lake without polling every row.
Mature answers mention replay: when you ship a bug in window logic, you re-read retained log or archive and recompute idempotent sinks, pipelines without replay are one-way glue.
Intuition
Batch mental model. Input is finite: orders_date=2024-11-09 partition. Job reads, aggregates, writes warehouse table daily_revenue. Failure = rerun job idempotently (INSERT OVERWRITE or merge key). Schedule: cron/Airflow at 02:00 UTC. Cost driver: scan bytes, not millisecond latency.
Stream mental model. Topic is infinite log; partitions (userId, merchantId) give parallel consumers; ordering guaranteed only per partition. Consumer group: each partition → one active consumer instance at a time. State for aggregations lives in RocksDB / changelog topic, replay rebuilds state if you reset offsets.
Windows. Tumbling (non-overlapping 5m), sliding (overlapping), session (gap-based). Late events: watermarks and allowed lateness, say if you drop or correct.
CDC. Debezium-style capture from DB WAL → stream → sink to warehouse/search. Schema evolution (ADD COLUMN) needs compatibility plan.
Semantics. Producers: at-least-once common. Consumers: idempotent sink (UPSERT by (userId, windowStart)), or transactional Kafka + idempotent producer when justified. Global total order across all users = single partition bottleneck, reject unless prompt requires it.
Lambda / kappa (one line). Maintaining separate batch and speed layers doubles code paths, modern trend is “stream with replay as batch” when retention allows.
Backpressure & lag. Consumer lag metric is the heartbeat; scale consumers when lag > SLO; consider autoscaling on lag derivative, not CPU.
Schema evolution. Avro/Protobuf registry; consumers tolerate unknown fields; reprocess when compatibility breaks.
What you will be able to do
- Choose batch vs stream from freshness needs
- Design a consumer group / partition story
- Plan replay when logic changes
When this comes up
- Kafka consumer group
- Windowed aggregation
- ETL
- Near-real-time pipeline
2See it
Architecture sketch
Log → consumers → sinks
Partitioned log retains history so you can replay after a bug.
Walkthrough
Replay after a bug
Idempotent sinks make history useful
1 / 4
Events append to a partitioned log. Ordering holds per partition key.
Read the walkthrough as text
Worked example, fraud alerts on payments + nightly finance report.
Events: PaymentAuthorized {userId, amount, ts, merchantId}; fraud rule: sum > $5k in rolling 5 minutes per user; finance needs daily totals by merchant.
- Split paths (interviewers love this): fraud = stream (SLA < 30s); finance = batch (SLA by 07:00).
- Stream setup: Kafka topic
payments, partition keyuserId, 32 partitions; Flink/Streams consumer groupfraud-rules. - Window: sliding or tumbling 5m sum
amountkeyed byuserId; state in embedded store; emit alert record when sum > threshold. - Sink idempotency: alert id
(userId, windowStart)upsert toalertstable, redelivery does not page twice. - Batch path: nightly Spark/ BigQuery scheduled query on
orderspartition, no need to share Flink cluster with fraud. - Bugfix story: deploy bad window boundary; replay from Kafka retention 7d or S3 archive; reset consumer group to T0; recompute alerts table with same idempotent keys.
- Ordering caveat: two events same user different partitions only if key wrong, always say “per-user order only.”
What interviewers listen for: explicit freshness SLAs; partition key choice; at-least-once + idempotent sink; batch for daily report; replay plan.
Napkin math
log = Kafka (partitioned append)
partitions ≥ consumer parallelism
key = userId → order within partition
consumer lag = end_offset − committed
retain 7 days → replay after bad deploy
Offset + replay beat fragile in-memory pipelines.
3Answer it
How to answer in an interview
When analytics, pipelines, “real-time dashboard,” or CDC appears:
- Freshness SLA, seconds vs hours → stream vs batch first.
- Source / sink, Kafka, S3, warehouse, ES; name partition key.
- Ordering scope, per-key only; reject global order unless required.
- Delivery semantics, at-least-once; idempotent sink pattern.
- State / windows if aggregating, tumbling/sliding in one sentence.
- Replay / backfill, retention, reset offsets, overwrite derived table.
- Anti-overengineering, no Kafka for weekly CSV if batch suffices.
Phrases: “partition-scaled parallelism,” “watermark for lateness,” “CDC from OLTP,” “recomputable derived view.”
- Capacity, partitions ≈ max parallel consumers; picking 1 partition caps throughput.
Answer checklist
Tradeoffs & numbers
- Stream: low latency; ops (lag alerts, consumer rebalance, schema registry), worth when SLA < few minutes.
- Batch: simpler/cheaper; stale until job completes, default for reporting.
- Long Kafka retention (30d): replay-friendly; storage $$$, tier to S3 for archive.
- Exactly-once Kafka transactions: stronger; latency and coupling, often idempotent sink enough.
- Lambda architecture: dual code paths, maintenance tax.
- When NOT to stream: executive PDF once daily; one-off migration; <1000 events/day.
- Ordering bugs: mis-keyed partition sends related events to different partitions, debug with keyed trace samples.
Kinds of questions (4)
Same pattern, different interview surfaces. Recognise which variant you are in before you answer.
Nightly batch ETL
Example. CFO needs merchant revenue by 07:00 CET; sub-minute freshness is irrelevant. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.
How. Partitioned warehouse job on `orders_dt`; idempotent overwrite; Airflow SLA alert if job misses window, no Kafka unless future real-time CFO dashboard appears.
Near-real-time aggregation
Example. Ops wallboard: orders per minute per region during Black Friday. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.
How. Kafka topic partitioned by `regionId`; tumbling 1m count; consumer scales with partitions; dashboard reads materialized view updated every few seconds.
CDC into warehouse
Example. OLTP schema changes weekly; analytics must stay in sync without nightly full dump. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.
How. Debezium on Postgres WAL → stream → BigQuery/Snowflake merge by primary key; schema registry for additive columns. Windowed aggregates per partition; checkpoint offsets; reprocess retained log on logic bugs.
Replay after bugfix
Example. Fraud model v2 fixed double-counting; last 48h alerts wrong. Choose batch vs stream from freshness SLO; partition key for parallelism; replay strategy after a bad deploy.
How. Reset consumer to `T-48h` offsets; idempotent `upsert` alerts table; keep v1 topic archive on S3 for longer backfill if retention short.
Common mistakes
Failure modes the AI coach watches for when you practise open questions.
Firehose without replay
Avoid. Direct service→service calls with no log. Bug means data is gone.
Prefer. Durable log + idempotent consumers + ability to recompute.
Weak sketch
Bad
Stronger sketch
Good
Stream for a daily report
Why it happens. Streaming stacks add brokers, consumer lag on-call, and schema ops when batch freshness already satisfies the business SLA.
How it fails. Proposal: Kafka + Flink cluster to generate a single daily compliance PDF attached to email at 6am.
Fix. Nightly partitioned batch ETL; add streaming only when a stated sub-minute or sub-hour SLA appears in requirements.
No replay
Why it happens. Derived tables encode logic; without reprocessing history, a bad deploy permanently poisons metrics until manual SQL surgery.
How it fails. Consumer commits offsets forward only; Kafka retention 24h; no archive, “we will fix forward.”
Fix. Retain log or export to object store; idempotent sinks; documented reset-offset backfill runbook.
Assumes global order
Why it happens. Parallel partitions interleave events; global ordering requires single partition and kills throughput.
How it fails. “Payment events must be processed in exact global order across all merchants on 64 partitions.”
Fix. Order per `accountId` partition key; design business rules for per-account sequencing only.
4Practise
Concept check
A short quiz on this lesson before open questions. Optional, and it does not gate practice.
Open questions
- about 8 min
Batch vs stream
Finance needs a daily revenue rollup emailed at 6am; operations wants a live “orders per minute” dashboard. For each requirement, choose batch or stream and justify freshness vs complexity.
- about 9 min
Windowed aggregation
Detect users spending more than $500 in any rolling five-minute window from a `payments` Kafka topic with 32 partitions. Sketch partitioning, consumer groups, and why global order is not required.
- about 8 min
CDC pipeline
Analysts want a warehouse copy of the orders table near real-time without polling Postgres every second. Describe CDC to a stream or warehouse sink and operational concerns (schema changes, lag).
- about 9 min
Replay after bug
A deployed stream processor miscounted refunds for three days before a fix shipped. How do you rebuild correct aggregates without losing Kafka messages, and what idempotency does the sink need?
Ready to practise?
Write an answer, get AI coaching, then revise. You are not building a real system.