In-Memory
Data Processing

Reading a megabyte sequentially from memory takes roughly 250 microseconds. Reading the same megabyte from a spinning disk takes roughly 20 milliseconds — about 80× slower. This isn't a Spark-specific number; it's one of the oldest, most-cited facts in systems engineering (traced to Jeff Dean and Peter Norvig's latency table), and it's the entire reason "keep it in memory" is a strategy at all rather than a marketing phrase.

Worked Example · Four Supersteps, Two Ways

Section 1.3's BSP shortest-path computation (Kitchen → Hub N/S/E → Zone Z) took 4 supersteps to converge. Between each pair of consecutive supersteps, the current best-known distances have to be handed forward — 3 handoffs in total for a 4-superstep job.

Four supersteps barely shows this — but a real iterative algorithm (PageRank over millions of edges, gradient descent over thousands of iterations) multiplies "3 handoffs" into "thousands of handoffs," and every one saved is another 80× avoided. Chapter 6's machine learning algorithms are exactly this kind of workload, which is precisely why they run on Spark rather than chained MapReduce in practice.

Practice

1. If Ch1.3's BSP job needed 4,000 supersteps instead of 4 (a much larger graph), how many disk round trips does the chained-job model pay, and how does that compare to the in-memory model?
   Solution

   3,999 disk round trips (one between each pair of consecutive supersteps) versus 0 for the in-memory model — the same ratio as the 4-superstep case, just with a thousand times more handoffs, meaning a thousand times more accumulated disk-round-trip cost for the chained-job approach specifically.

2. A job reads its input from HDFS exactly once, does one map and one reduce, and writes its output once. Does the memory-vs-disk distinction from this section meaningfully change its runtime?
   Solution

   Barely — there's no intermediate handoff to avoid. This section's savings come specifically from avoiding repeated disk round trips between chained steps; a single-pass job was never paying that cost more than once, so there's little for in-memory processing to eliminate.

3. Using the 250µs-vs-20ms figures, roughly how many memory-speed handoffs could happen in the time ONE disk-speed round trip takes?
   Solution

   Roughly 80 — 20ms ÷ 250µs = 80. In the time a single chained-job disk hop completes, an in-memory pipeline could have already finished roughly 80 equivalent handoffs.

4. Section 3.6 noted that Spark's lineage-based recomputation can cost MORE than a MapReduce task retry if a long chain has to be replayed uncached. Does that observation weaken this section's case for in-memory processing?
   Solution

   Not really — it's a distinct trade-off. This section is about the cost of the HAPPY PATH (no failures) for repeated handoffs; Section 3.6 is about the cost of the UNHAPPY PATH (a failure occurs) for recomputation. In-memory processing can be strictly faster when everything succeeds and still have a recovery cost worth planning for when something doesn't — both things are true at once, which is exactly why Section 4.3 treats caching as a bet rather than a free win.

Section 3.5 was precise about laziness: nothing runs until an action is called. What it didn't dwell on is what happens when multiple actions are called on the same chain of transformations. Without intervention, Spark reruns the whole chain from source, every single time — caching is the intervention.

Worked Example · Three Reports, With and Without

See Both DAGs

Practice

1. A fourth report is added, also built on the same veracity-filtered 15 rows. With caching already in place from the worked example, how many additional disk reads does this fourth report need?
   Solution

   Zero — it reads from the same cached result Reports 2 and 3 used. Caching's benefit scales with the number of consumers of the cached data; a fourth consumer is free, just like the second and third were.

2. If only Report 1 ever runs, and Reports 2 and 3 are hypothetical future additions that haven't been written yet, was caching before Report 1 worth it?
   Solution

   No — with only one consumer, caching added the cost of storing the 15 rows with zero reuse to offset it. Caching should be added when a shared computation actually has multiple consumers, not speculatively "just in case" one shows up.

3. An RDD is cached with the default level, and its data doesn't fit entirely in the available memory. What happens to the partitions that don't fit, and what happens the next time they're needed?
   Solution

   Under MEMORY_ONLY (the RDD default), partitions that don't fit are simply not cached at all — not an error, just silently absent. The next time they're needed, Spark recomputes them from lineage (Section 3.6), exactly as if they'd never been marked for caching in the first place.

4. Section 4.1 showed avoiding disk round trips between BSP supersteps. Is caching the SAME mechanism as keeping an RDD "in memory across supersteps," or a different one?
   Solution

   Same underlying mechanism, different trigger. Section 4.1's supersteps pass results forward automatically as part of one connected computation. Caching is the same "keep it in memory" idea applied deliberately, by the programmer, to a specific RDD that will be reused by multiple separate later actions rather than passed forward once within a single DAG.

Memory is finite in a way disk usually isn't. When a new cached result needs room and none is free, Spark evicts something — using LRU (Least Recently Used): whichever cached block hasn't been touched in the longest time goes first, on the theory that the least recently used data is the least likely to be needed next.

Worked Example · Three Results, Two Slots

Step Through the Cache

Practice

1. Redo the worked example, but skip step 3 (no re-access of A before caching C). Which result gets evicted now?
   Solution

   A — without the step-3 refresh, A and B were cached in that order (A first), making A the least recently used when C needs room. Whichever of the two was touched longest ago is what LRU removes; the refresh in the original trace is precisely what changed the outcome.

2. With 3 memory slots instead of 2, does caching A, B, and C in sequence cause any eviction at all?
   Solution

   No — three results fit in three slots with room to spare. Eviction only happens when a new cached result needs space and none is free; enough capacity for every live cached result removes the pressure entirely.

3. Under MEMORY_AND_DISK instead of MEMORY_ONLY, does step 4 of the worked example still evict something, or does the eviction itself go away?
   Solution

   Eviction from the memory tier still happens — there are still only 2 memory slots. What changes is the CONSEQUENCE: B spills to disk rather than disappearing, so step 5's "cache miss" becomes a disk read instead of a full recompute from source.

4. Section 2.1 discussed the NameNode's re-replication after a DataNode failure. Is an evicted cache entry's "recomputation via lineage" the same kind of repair, or a different kind?
   Solution

   Different kind, same underlying idea of "restore what's missing on demand." Section 2.1's re-replication restores a DURABLE guarantee (a replicated block) after an actual failure, coordinated by the NameNode without being asked. Cache recomputation restores a PERFORMANCE optimization (a cached result) after a routine, expected memory-management decision, triggered only when something later asks for the missing result — nothing proactively "repairs" an evicted cache entry the way the NameNode proactively repairs a lost block.

An RDD is a set of instructions Spark executes essentially as written. A DataFrame (or a Dataset, or a SQL query) is a step higher: a structured, schema-aware description of what you want, which Spark's Catalyst optimizer is free to rewrite into a more efficient how before anything runs.

Worked Example · Filtering to Hub E, Two Ways

Recall Section 2.6: if CityCourierDB partitions its storage by hub, Hub E's 5 orders (#4, #7, #10, #13, #16) live entirely in one shard, separate from Hub N's and Hub S's 11.

Both return the identical 5 rows. The RDD version paid for reading 16; the DataFrame version paid for reading 5 — the exact "1 node instead of 3" saving Section 2.6 measured for hub-partitioned selective queries, now happening automatically because Catalyst could see and rewrite the query, instead of manually because a programmer chose the right partitioning-aware code path by hand.

Practice

1. If CityCourierDB instead partitioned by hash-of-order-id (Section 2.3) rather than by hub, would the DataFrame filter in the worked example still skip 2 of 3 shards?
   Solution

   No — Section 2.3 and 2.6 already established that hash-of-order-id partitioning scatters each hub's rows across multiple shards with no relationship to the hub field. Catalyst can only push a predicate down to skip partitions when the partitioning scheme actually aligns with the filtered column; under hash partitioning, all 3 shards would still need reading regardless of DataFrame vs RDD.

2. A DataFrame query filters on a column wrapped inside a custom user-defined function: .filter(myCustomCheck(df.hub)). Does Catalyst's pushdown still apply?
   Solution

   Generally no — a UDF is opaque to Catalyst, just like an RDD's arbitrary function. The optimizer can't see inside myCustomCheck to know it's equivalent to a simple equality check on a partitioned column, so it falls back to reading everything and filtering in Spark, exactly the RDD-style cost.

3. Combine this with Section 4.2: does caching a DataFrame after Catalyst has already applied predicate pushdown still make sense for repeated queries?
   Solution

   Yes, and the two stack: pushdown reduces what gets read on the first pass (5 rows instead of 16); caching that already-filtered 5-row result means a second identical query reads 0 rows from anywhere, not even the pushdown-optimised 5. They solve different problems — one shrinks a single read, the other eliminates repeat reads entirely.

4. A team writes their whole pipeline in raw RDDs "for full control," on hub-partitioned storage, running the same Hub E filter thousands of times a day. What's the accumulated cost of skipping DataFrames, in this section's terms?
   Solution

   Every one of those thousands of runs reads all 16 rows (all 3 shards) instead of the 5 a DataFrame query would read via automatic pushdown — the "full control" bypassed the one optimization (Catalyst) that would have exploited the partitioning scheme's own design without requiring any manual partition-aware code at all. The RDD API isn't wrong to use, but it trades this specific automatic optimization away.

1. An RDD is cached but only ever consumed by one action, once. Was caching it worth the memory it used?
   Solution

   No — caching pays off across multiple consumers of the same computation. A single consumer gets no reuse benefit, only the cost of storing the result.

2. What's the default storage level behind a plain .cache() call on an RDD, and what happens to a partition that doesn't fit?
   Solution

   MEMORY_ONLY. A partition that doesn't fit is simply not cached — not an error — and gets recomputed from lineage the next time it's needed.

3. A cached DataFrame's partition gets evicted under MEMORY_AND_DISK. Where does Spark look for it next time, before falling back to full recomputation?
   Solution

   Local disk first — MEMORY_AND_DISK spills evicted blocks to disk rather than dropping them outright, so the next access reads from disk (Section 4.1's slower but far cheaper-than-recompute rate) instead of rebuilding from source.

4. Why doesn't Catalyst's predicate pushdown help a query filtering on a column wrapped in a user-defined function?
   Solution

   Because a UDF is opaque to the optimizer — Catalyst can't inspect what it does, so it can't prove the filter is safe to push down to storage. It falls back to reading everything and filtering afterward, same as plain RDD code.

5. Section 4.1's memory-vs-disk ratio is about 80×. Does that mean an in-memory Spark job is always 80× faster than an equivalent chained-MapReduce job?
   Solution

   No — 80× is the ratio for one hop of comparable size. Real speedup depends on how many disk round trips were actually avoidable and how large each was; a job with few or no chained steps has little disk-hop cost to remove in the first place.

6. LRU evicts block B to make room for block C. Two minutes later, B is requested again and C is not. Did LRU make the right call, looking back?
   Solution

   No, in hindsight — but LRU only has past access patterns to go on, not future ones. It bet that recency predicts near-future use, and this particular case went against that bet, which can always happen with any recency-based heuristic.

7. Is reading the original order log from Chapter 2's replicated HDFS blocks affected by anything in this chapter?
   Solution

   No — this chapter is entirely about what happens to INTERMEDIATE results after the first read: caching them, evicting them, or letting Catalyst skip reading irrelevant parts of them sooner. The durable, replicated source data and its storage mechanics are untouched, exactly Section 4.1's "still reads from disk at least once" note.

8. A team caches a DataFrame, then writes the rest of their pipeline using only raw RDD transformations on top of it. Do they still get Catalyst's pushdown benefits for those later RDD steps?
   Solution

   No — once the pipeline drops to the RDD API, Catalyst is no longer in the picture for those steps, regardless of what happened before. Caching and Catalyst optimization are separate mechanisms; using one doesn't guarantee the other applies to code that bypasses the API it depends on.

• Spark: The Definitive Guide — Chambers & Zaharia. The authoritative treatment of caching, storage levels, and the Catalyst optimizer's rewrite rules behind every section of this chapter.
• Designing Data-Intensive Applications — Martin Kleppmann. Its chapters on batch and stream processing frame why in-memory intermediate state matters beyond Spark specifically, useful context before Chapter 5's streaming material.
• Hadoop: The Definitive Guide — Tom White. Read alongside this chapter for the contrast: MapReduce's original disk-backed-everything design is the baseline this chapter's savings are measured against.