Streaming
Analytics

A batch job can average "everything" because everything has an end. A stream doesn't, by definition — so any aggregate over a stream has to first answer which slice of the unbounded flow it's averaging. A window is that slice, and there are three standard shapes.

Worked Example · The Same Stream, Three Ways

See All Three, Same Timeline

Practice

1. With a 300s tumbling window, which single window does order #10 (Section 1.1's veracity bug, t=378) fall into?
   Solution

   [300,600) — 378 sits between 300 and 600. Exactly one window, since tumbling windows never overlap.

2. Using the 300s/150s sliding window scheme, list every window order #10 (t=378) appears in.
   Solution

   [150,450) and [300,600) both contain 378. Two windows, because sliding windows with a 150s step and 300s size overlap by exactly half their width — any event lands in exactly two consecutive windows (except near the very start or end of the stream).

3. If the dinner rush's first order arrived at t=1,000 instead of t=1,900 (a gap of 352s from the lunch rush's last order at t=648), would the 600s session threshold still split it into two sessions?
   Solution

   No — 352s is under the 600s threshold, so the dinner order would extend the SAME session as the lunch rush rather than starting a new one. Session boundaries are entirely about whether the gap exceeds the threshold, not about any fixed clock time.

4. Section 1.5 showed a 2-node split being less efficient than 3 nodes due to imbalance. Does a sliding window's overlapping computation create a similar kind of "wasted work," and is it avoidable the same way?
   Solution

   It's a related but distinct cost — sliding windows deliberately recompute overlapping regions (orders 5–8 processed twice in the worked example) as the entire point of the mechanism, not an accidental imbalance to fix. Section 1.5's imbalance was solvable by rebalancing the split; a sliding window's overlap is inherent to getting frequent updates and can only be reduced by choosing a larger step (at the cost of less frequent updates), not eliminated while keeping the same freshness.

Every streamed event carries (at least) two timestamps, and streaming systems that don't distinguish them get silently wrong answers under exactly the conditions Chapter 1's CAP discussion already flagged as normal.

Worked Example · Two Delayed Orders, One Boundary Crossed

Hub S loses connectivity from t=260 to t=350 (Section 1.6's 90-second partition). Orders #8 (Wrap City, event time 289) and #9 (Curry Point, event time 333) are placed locally during the outage and only reach the stream processor once Hub S reconnects at t=350.

Recomputing the delivery-time averages from Section 1.3 under each rule:

Both numbers in every cell changed — not just for order #8's own window, but for window 1 too, since it silently gained a record that never belonged there. Nobody has to make an obvious mistake for this to happen; it happens automatically the moment a system windows by "when I saw it" instead of "when it happened," under conditions (a network hiccup) that Chapter 1 already established are routine, not exceptional.

Practice

1. If Hub S's outage instead ran from t=550 to t=640, which orders would be delayed, and would either cross a tumbling window boundary?
   Solution

   Orders with event time in [550,640): order #14 (t=556) and order #15 (t=602). Both have event time within [300,600) and [600,900) respectively — if both are processed once Hub S reconnects at t=640, order #15's processing time (640) still falls in its own true window [600,900), no crossing. Order #14's event time (556) is in [300,600), and if processed at t=640 that's already in [600,900) — a crossing, structurally identical to order #8's case.

2. Recompute window 1's [300,600) naive processing-time average if order #14 (delivery time 750) is the one wrongly excluded (moved to window 2) instead of order #8.
   Solution

   Correct window 1 (event time) is {9,10,11,12,13,14}, avg 570.0. Removing #14 leaves {9,10,11,12,13}: (600+(−60)+930+510+690)/5 = 2670/5 = 534.0. A different order crossing a different boundary shifts the average differently — the size and direction of the error depends on which specific record got misplaced.

3. A stream processor timestamps every event with processing time only, no event time at all. Can Section 5.1's tumbling windows still be computed?
   Solution

   Yes — they'd just be processing-time windows by definition, not a "wrong" version of event-time windows. The problem in this section isn't that processing-time windows are invalid; it's that USING processing time while BELIEVING the result reflects event time is the error. A system that's honest about which clock it's using hasn't made a mistake, even if event time would have been more meaningful for the question being asked.

4. Chapter 3 distinguished a task's retry cost from a job-wide restart. Is order #8's boundary-crossing error a "retry" problem in that sense?
   Solution

   No — nothing failed and nothing is being retried. Order #8 was processed exactly once, successfully, just under the wrong window assignment. This is a correctness problem in how time is interpreted, entirely orthogonal to Chapter 3's fault-tolerance mechanics, which assume the question "which window does this belong to" was already answered correctly and only worry about task completion.

A watermark is a stream processor's running, heuristic claim: "I have now seen every event with event time before T." It's a bet, not a guarantee — but it's the mechanism that lets a window ever actually close and emit a result, rather than waiting indefinitely for events that might never arrive.

Worked Example · Order #8, Two Policies

Order #8 (event time 289, processing time 350) is 50 seconds late relative to window 0's boundary at t=300.

Notice the 40s policy's n=7 result is numerically identical to Section 5.2's "naive processing-time" mistake — same number, completely different status. The naive version silently misplaced order #8 into the wrong window, believing its answer was complete. The watermark version deliberately excludes it, knows it's incomplete, and can say so.

Watch the Watermark Decide

Practice

1. Under a 60s allowed-lateness policy, at what processing time would window 1 [300,600) become eligible to emit?
   Solution

   t = 600 + 60 = 660. The same rule applies to every window: eligible once the watermark passes the window's own end plus the allowed-lateness constant.

2. With allowed lateness = 60s, is order #9 (processing time 350, event time 333, window 1) included in window 1's result regardless?
   Solution

   Yes, comfortably — window 1 doesn't even become eligible to emit until t=660, and order #9 arrives at t=350, hundreds of seconds before that. This is exactly Section 5.2's point: order #9's delay never approached being a problem for any reasonable policy.

3. A team sets allowed lateness to 0 seconds, believing it gives the "most correct" result fastest. What actually happens to order #8 under this setting?
   Solution

   Window 0 becomes eligible to emit the instant the watermark passes t=300 — before order #8 (arriving at t=350) has any chance of inclusion, no matter what. Zero allowed lateness doesn't mean zero lateness tolerance is achieved for free; it means the window commits to whatever it has the moment the clock hits the boundary, guaranteeing any delayed data is excluded.

4. Connect this to Section 4.3's cache eviction: is "deciding a window is closed" the same kind of bet as "deciding to evict the least-recently-used cache entry"?
   Solution

   Structurally similar, different stakes. Both are heuristic decisions made without complete information (recency as a proxy for future use; watermark progress as a proxy for "no more data coming") that can turn out wrong in hindsight. The difference is recoverability: an evicted cache entry can simply be recomputed via lineage with no correctness impact (Section 3.6). A window that emitted too early has already published a number someone may have acted on — correcting it means either a retraction or living with the error, a strictly costlier kind of "wrong bet."

A message broker like Kafka decouples producers of events from consumers of them: hubs publish orders without knowing who's listening, and the stream processor reads them without knowing (or caring) which hub sent which. The unit of organisation is a topic — and every topic is split into partitions for exactly the same reason Chapter 2 split files into blocks: parallelism.

Worked Example · The "orders" Topic, Partitioned by Hub

Practice

1. If the "orders" topic had 5 partitions instead of 3, keyed by restaurant instead of hub, how many orders would each partition hold (Section 3.2's restaurant counts)?
   Solution

   Curry Point: 4. Momo Hut: 3. Wrap City: 3. Sushi Go: 3. Pizza Barn: 3 — five partitions of 4,3,3,3,3, matching Section 3.2's MapReduce count-per-restaurant results exactly, since both are grouping the same 16 rows by the same key.

2. A consumer in the group crashes and restarts. Are the events in its assigned partition lost?
   Solution

   No — the partition itself is a durable, replicated log (Kafka partitions are typically replicated across brokers, the same durability principle as Section 2.1's HDFS blocks). The restarted consumer resumes from its last committed offset in that partition; nothing about the consumer failing deletes anything from the log itself.

3. Two separate consumer groups both subscribe to the "orders" topic — one computing live revenue, one archiving to cold storage. Does the revenue consumer's processing speed affect what the archiving consumer sees?
   Solution

   No — separate consumer groups track their own offsets independently against the same durable log. Kafka's log-not-queue design (this section's note) means one group's read speed or backlog has zero effect on any other group's ability to read the identical events from the beginning.

4. Section 2.3 showed that hash-of-order-id partitioning scatters a single hub's rows across every storage node. If "orders" were instead partitioned by hash-of-order-id rather than by hub, would a consumer assigned to "Hub E's data" even be a coherent idea?
   Solution

   No — exactly Section 2.3's lesson, replayed at the streaming layer. Hash-of-order-id partitioning has no relationship to hub, so Hub E's orders would be scattered across all 3 partitions; no single consumer could claim "Hub E's stream" without reading and filtering every partition, the same 3-nodes-instead-of-1 cost Section 2.6 measured for storage.

Spark Structured Streaming's engine is fundamentally the same batch engine Chapter 3 described: every micro-batch is a genuine, small Spark job, complete with its own ApplicationMaster negotiation (Section 3.1) and DAG scheduling (Section 3.5), just triggered automatically on a timer instead of by a person running a script. Flink was built the other way around: continuous processing is the native model, and a "batch" is just the special case of a stream that happens to end.

Worked Example · The Same Tumbling Window, Two Engines

Both approaches produce the identical final answer for a completed window. They differ in how current the in-progress answer is at any given moment before the window closes — and in how much scheduling overhead (Section 3.1's container negotiation, repeated for every micro-batch) the system pays along the way.

Practice

1. If the micro-batch trigger interval were shortened from 10 seconds to 1 second, does Spark's architecture become identical to Flink's?
   Solution

   No — it becomes 10× more micro-batches, each still a small discrete job with its own scheduling overhead, just more frequent. Shrinking the interval narrows the latency gap but doesn't change the fundamental "many small batch jobs" architecture into "one continuous pipeline."

2. A dashboard needs to update no faster than once every 5 minutes. Does this section's latency comparison matter for choosing a framework here?
   Solution

   Barely — both Spark's seconds-scale micro-batch latency and Flink's tens-of-milliseconds are far below the 5-minute requirement. Framework choice here would more reasonably turn on other factors (ecosystem, existing skills, operational familiarity) than on this section's latency numbers.

3. Section 5.4 noted Kafka partitions cap useful consumer parallelism. Does that ceiling apply differently to a Spark micro-batch job versus a Flink streaming job reading the same topic?
   Solution

   Same ceiling either way — the partition count bounds how many parallel readers can exist regardless of what happens after reading. A Spark micro-batch job's tasks and a Flink job's parallel operator instances both ultimately map to Kafka partitions as the unit of read parallelism, exactly Section 5.4's rule, independent of which processing architecture consumes them.

4. Connect this to Section 3.6: does a failed micro-batch job get retried the same bounded way a MapReduce task does?
   Solution

   Yes — a failed micro-batch is a failed small Spark job, and Section 3.1–3.6's ApplicationMaster-and-task machinery applies to it exactly as it would to any other Spark job. A true-streaming system like Flink instead relies on periodic checkpoints of its continuously running state, a different mechanism aimed at the same goal: bounded recovery cost after a failure, not a full restart from the beginning of the stream.

Section 5.4 established Kafka's default as at-least-once delivery: a consumer commits its offset after processing, so a crash between "processed" and "committed" means the same message gets redelivered. This is a deliberate trade, not an oversight — the alternative, committing before processing, risks silently losing a message if the crash happens the other way round.

Worked Example · Double-Counted, Then Fixed

Practice

1. A consumer counts orders (not revenue) using "increment a counter by 1 per message." Is this operation idempotent under redelivery?
   Solution

   No — incrementing by 1 for every delivery double-counts a redelivered message exactly like the naive revenue example. Making it idempotent requires the same fix: key by order ID (e.g., "has this order ID already been counted?") rather than blindly incrementing on every message received.

2. Section 5.3's watermark-based window emission already published window 0's result. If order #8 is later redelivered due to an at-least-once retry, does idempotent processing fix the fact that the window already closed?
   Solution

   No — these are two separate problems. Idempotency prevents order #8 from being double-counted if it's redelivered; it does nothing about whether order #8 was ever included in the right window in the first place. A late arrival after a window has closed still needs Section 5.3's late-data policy (drop, side-output, or retract), regardless of how cleanly the delivery layer handles duplicates.

3. Would switching Kafka to at-most-once delivery (commit before processing) have avoided this section's double-counting problem?
   Solution

   Yes, but by trading it for the opposite failure: a crash between committing and processing would now silently lose order #8 entirely, with no retry to recover it. At-most-once avoids duplication by accepting loss; the worked example's fix (idempotent processing under at-least-once) avoids both.

4. Chapter 3 distinguished a bounded task retry from a full job restart. Is order #8's redelivery here the streaming equivalent of a "task retry," in the same bounded sense?
   Solution

   Yes — exactly one message gets reprocessed, not the whole stream from the beginning. Kafka's offset-based redelivery is bounded to whatever wasn't committed, the streaming analogue of Section 3.6's bounded task retry: both systems retry the smallest unit of work that might have been lost, not everything that came before it.

1. An event's timestamp is set by the stream processor at the moment it's read, not by the source device. Is this event time or processing time?
   Solution

   Processing time — event time requires the timestamp to reflect when the thing actually happened at its source, not when the system happened to observe it.

2. A Kafka topic has 4 partitions. A consumer group has 6 consumers. How many sit idle?
   Solution

   2 — each partition is read by exactly one consumer in a group at a time, so only 4 of the 6 consumers can be doing anything at all.

3. A sliding window has size 10 minutes and step 10 minutes. Is this actually a sliding window, or something else?
   Solution

   It's functionally a tumbling window — when step equals size, there's no overlap between consecutive windows, which is the defining property of tumbling, not sliding. Sliding specifically requires the step to be smaller than the size.

4. A window's watermark passes its close threshold and the window emits a result. Three seconds later, a legitimately late event for that window arrives. What are a system's realistic options?
   Solution

   Drop it, route it to a side output for separate handling, or issue a retraction that corrects the already-published result. Silently and invisibly folding it into a different window (Section 5.2's naive-processing-time mistake) is not a real option, since it produces a specific, silent kind of wrong answer.

5. A payment-processing system chooses at-most-once delivery for a "send confirmation email" step. What failure mode does this accept, and why might that be the right call here?
   Solution

   It accepts that a confirmation email might occasionally never be sent (silent loss) rather than risk sending the same email twice. For a low-stakes notification, an occasional missed email is often more tolerable than annoying or confusing a customer with duplicates — a reasonable trade in a way it would NOT be for the payment charge itself.

6. Spark Structured Streaming's micro-batch architecture is described in this chapter as reusing Chapter 3's machinery. Which specific Chapter 3 concept does each micro-batch correspond to?
   Solution

   A single Spark job — complete with its own ApplicationMaster negotiation (Section 3.1) and DAG scheduling (Section 3.5), just triggered automatically on a timer rather than by a person.

7. A session window's gap threshold is set far too low for the actual data pattern. What's the likely symptom?
   Solution

   Far too many sessions — ordinary pauses between related events (that should belong to one session) would exceed the threshold and incorrectly split a single real session into several fragments.

8. Does idempotent processing, by itself, solve the event-time-vs-processing-time problem from Section 5.2?
   Solution

   No — they're unrelated problems. Idempotency ensures a redelivered message doesn't get double-counted; it says nothing about which window a message's timestamp should place it in. A system can be perfectly idempotent and still window entirely by the wrong clock.

• Streaming Systems — Akidau, Chernyak & Lax. The definitive treatment of event time, watermarks, and windowing strategies behind Sections 5.1–5.3 — the book this chapter's further-reading note in Chapter 1 was pointing toward all along.
• Designing Data-Intensive Applications — Martin Kleppmann. Its stream-processing chapter covers Kafka's log-based design (Section 5.4) and exactly-once semantics (Section 5.6) with more depth on the underlying replication mechanics.
• Spark: The Definitive Guide — Chambers & Zaharia. Covers Structured Streaming's micro-batch model (Section 5.5) as an extension of the same DAG engine Chapter 3 and Chapter 4 already introduced.