Big Data
Fundamentals

Sixteen rows is not big data by any definition. You can read all of them in the time it takes to scroll. But every large-scale system this course studies is built to survive the moment a table like this stops being sixteen rows and becomes sixteen billion — and the only way to see what breaks is to know, in complete and boring detail, what the small version looks like first.

The five things that get harder, not the one number that gets bigger

It is tempting to define "big data" as "data too big for one computer." That definition is not wrong, but it hides the useful part. Data resists being handled by one machine for at least five independent reasons, and a system that only solves one of them is not a big-data system — it is a bigger spreadsheet.

These five are independent axes, not stages of one problem. A dataset can be enormous and still perfectly trustworthy (a well-run sensor archive). A dataset can be tiny and still unusable (three rows, all wrong). The challenges CO1 asks you to analyze — storage, processing, heterogeneity, quality, extraction of value — map onto these five almost one-to-one: Volume and Velocity drive storage and processing challenges; Variety drives heterogeneity; Veracity drives quality; Value is the challenge of the other four actually paying off.

Worked Example · The CityCourier Log

Here is the entire spine dataset for this chapter: every order CityCourier's three hubs handled during one fifteen-minute lunch rush. Read it once, in full, the way you would a printed table — you will not get this chance again once the chapters ahead make it bigger.

Walk the five V's across this one table:

• Volume — 16 rows at rest. Small here; the "at scale" tab above already showed the same rate holds 1,536 orders a day per hub-set, ~46,000 across a city's worth of hub-sets. The row count is the least interesting axis — it is also the only one a bigger hard drive can fix.
• Velocity — the 16 orders span 636 seconds, so a new order lands roughly every 42.4 seconds on average (636 ÷ 15 gaps). A system that can only store orders but not absorb one every 42 seconds without falling behind has a velocity problem even if it never runs out of disk.
• Variety — this table shows only the structured fields. The same order also carries a GPS breadcrumb trail (semi-structured, Section 1.6) and, in the real app, a free-text review. Three shapes, one order.
• Veracity — three ratings are missing (18.8% of the log) and order #10 claims a delivery time of −60 seconds: a courier who arrived a minute before the food left the kitchen. That second problem is worse than the first. A missing value announces itself; a wrong value that is still a plausible-looking number does not.
• Value — none of the above matters until someone asks a question of it. "Which hub needs another rider?" turns this table into a staffing decision. Nobody asking anything turns it into disk usage.

Interactive · One Bad Record, One Silent Number

Practice

1. Using the full log, count how many orders each hub handled. If CityCourier can afford exactly one additional rider today, which hub should get them?
   Solution

   Hub N: 6 orders (#1,3,6,9,12,15). Hub S: 5 orders (#2,5,8,11,14). Hub E: 5 orders (#4,7,10,13,16). Total 16, which checks out against the log. Hub N is one order ahead of the other two, so the marginal rider goes there first.

2. Suppose you learn that order #7's rating of "—" is not missing at all: the review system logged a genuine 0-star rating as blank during a five-minute app outage, and 0 is a valid score in this app. Does reclassifying #7 change which V category its problem belonged to, and does it change the missing-rating percentage?
   Solution

   Yes to both. #7 moves out of Veracity (a value that shouldn't be trusted) because there is now nothing wrong with it — it was a Variety/ingestion bug (the outage encoded a valid low score as an absent one), already fixed by knowing the true value. The missing-rating count drops from 3/16 (18.8%) to 2/16 (12.5%), since #3 and #11 are still genuinely blank.

3. A dashboard reports CityCourier's average rating as 4.0, computed from the 13 non-missing scores, and a manager concludes "customers love us." What's wrong with that conclusion — assuming the arithmetic itself is correct?
   Solution

   The conclusion assumes the three missing ratings are missing for reasons unrelated to how the order went — that they are, in the jargon, missing completely at random. There is no evidence for that here, and reason to doubt it: if customers who had a bad experience are less likely to bother rating at all, the missing three are disproportionately the unhappy ones, and 4.0 overstates satisfaction. Veracity is not just "is the visible data correct" — it's also "is what's invisible invisible for an innocent reason."

4. If CityCourier's order volume grows 100× next year on the same order-entry app, will the rate of impossible records like #10 improve, worsen, or stay the same — and separately, will the count of them stay manageable?
   Solution

   The rate stays the same: whatever software defect produced a negative delivery time is a property of the app, not of how many orders flow through it, so scaling Volume alone doesn't touch Veracity's underlying cause. But the count scales with volume — one bad row in sixteen becomes roughly a hundred in sixteen hundred, thousands per day at city scale. A rate you could catch by reading the table stops being catchable by eye long before it stops being small; this is precisely why Chapter 3 onward builds automated data-quality checks into the pipeline instead of leaving them to a human glance.

Follow order #4 — Hub E, Sushi Go, ₹560, the most expensive order in the log. At t = 130s it is a tap on a customer's phone. Twenty-four hours later it may be three different things at once: a settled row in a ledger, an input to tomorrow's staffing recommendation, and (per Section 1.7's retention rules) a countdown toward eventual deletion. A large-scale system has to manage all three simultaneously, for every row, forever — which is why the data life cycle is treated as a first-class design object, not an afterthought to storage.

Seven stages, and a hand-off at every arrow

Every large-scale system moves a record through roughly the same seven stages, whatever the domain:

Each arrow between stages is a network hop, and every network hop is a place a record can be delayed, duplicated, or lost — which is exactly the boundary Section 1.6 spends its cost-trade-off discussion on. The life cycle is not a diagram you draw once; it is the map of every place your system can fail.

Worked Example · Order #4, Stage by Stage

The Life Cycle, Drawn

Practice

1. List, in order, the seven life-cycle stages order #9 (Hub N, Curry Point, ₹355) passes through between the customer's tap and the moment it appears inside a monthly revenue report.
   Solution

   Generate (tap) → Collect (edge capture at Hub N) → Ingest (validated, timestamped, queued) → Store (written durably to Hub N's log) → Process (folded into a batch aggregate) → Analyze/Visualize (appears in the monthly report) — with Archive/Purge still ahead of it, on the same 90-day / 2-year schedule as every other row.

2. CityCourier's legal team rules that GPS breadcrumb data must be purged 24 hours after delivery, while the order record itself (amount, rating) is kept for 2 years for tax purposes. Does a single order therefore have one life-cycle length or several?
   Solution

   Several — the life cycle is defined per field, not per row, once retention policy gets involved. The GPS trail for order #9 completes its full seven stages, including purge, within a day; the transactional fields of the same logical order are still in the Store/Archive stages nearly two years later. A "record" in governance terms is really a bundle of independently aging fields.

3. A customer exercises their right to be forgotten. Support confirms the order is deleted from the primary orders database. Is CityCourier's obligation now satisfied?
   Solution

   Not necessarily. The note above lists at least three other places the same record could still exist: a backup taken before the delete, a replica at another hub not yet caught up, and any batch-layer aggregate (Section 1.4) computed while the row still existed. Deleting from one store proves the life cycle's Store stage was touched, not that every downstream stage was.

4. Order #10's impossible −60-second delivery time (Section 1.1) was actually detectable the moment the courier's app reported it. At which life-cycle stage should that check ideally happen, and why does catching it at the Process stage instead (as a nightly batch job did, in this trace) cost more?
   Solution

   Ideally at Ingest, where the record is still one hop from its source and can be rejected, flagged, or sent back for correction before it is durably stored anywhere. Catching it at Process means the bad value has already been replicated, already sits in at least one archive snapshot, and any aggregate computed before the nightly job ran (a dashboard refresh, a speed-layer estimate) already absorbed the bad number. The cost of a data-quality bug grows with every life-cycle stage it survives.

Every model in this section answers the same underlying question — how do independent machines cooperate without sharing memory? — but each answers it under different constraints: is the input finite or endless? Does step 2 need step 1's global result, or just its own slice? These constraints, not preference, are what determine which model fits a given problem.

The foundation: shared-nothing

CityCourier's three hubs already are a shared-nothing architecture: each has its own CPU, its own memory, its own local order log, and the only way one hub learns anything about another is by sending it a message over the network. No hub can reach into another's memory. This is a deliberate design, not a limitation to work around — it is what lets CityCourier add a fourth hub next year by buying one more machine, instead of buying a bigger one.

Given that foundation, four models answer "how do the messages flow?" differently:

Worked Example · MapReduce, Counting Orders Per Restaurant

Ask "how many orders did each restaurant get?" and MapReduce's two phases fit exactly: each hub can count its own restaurant totals with zero coordination (the map), and the only cross-hub work is merging same-named restaurants together (the reduce).

Notice what never had to happen: no hub ever asked another hub "what have you got so far?" mid-map. That independence is exactly why MapReduce scales — and exactly why it struggles with the next question.

Worked Example · BSP / Graph Processing, Shortest Route to Zone Z

Now ask a different kind of question: "what's the fastest route from the Kitchen to Zone Z, a customer cluster reachable only through the hubs?" This is not embarrassingly parallel — Hub E cannot know its true distance from the Kitchen until it has heard from whichever hub sits between them, which itself had to hear from the Kitchen first. The answer has to propagate, in rounds. That is precisely what BSP is for, and graph algorithms are its most common use — Google's Pregel, the model this course's "graph processing" bullet ultimately refers to, is BSP with one rule added: think like a vertex. Every node runs the same tiny program, knowing only its own state and its neighbours' messages.

Worked Example · Stream Processing, a Running Average That Doesn't Wait

Last question: "what's the average delivery time, updated continuously, without waiting for the lunch rush to end?" Batch models assume you can see the whole input before answering; a stream model assumes you never will. The usual fix is the tumbling window: chop the endless stream into fixed, non-overlapping slices — here, 300 seconds each — and emit one average per slice as soon as it closes.

grey = every reading in the window as it arrived · amber = veracity-filtered (Section 1.1's rule applied per window)

Window 2 (300–600s) is the one to stare at: its raw average, 570.0s, is lower than its filtered average, 696.0s — because order #10's impossible −60-second reading was pulling the raw number down, not up. A stream engine that reports 570.0s without a veracity check isn't just imprecise, it's reporting the wrong direction of the story: deliveries in that window were actually slower than average, not faster.

Practice

1. Run the same map/shuffle/reduce pattern shown above, but group by hub instead of restaurant. What are the three reduced counts, and do they sum to 16?
   Solution

   Map: each order emits (hub, 1) — this needs no shuffle at all, since every order already lives on its own hub's node; the "grouping" is free. Reduce: Hub N → 6, Hub S → 5, Hub E → 5. Sum: 6+5+5 = 16, matching the log's row count.

2. Suppose the Hub S → Zone Z edge weight were revised from 8 minutes to 3 (a new shortcut opens). Recompute the shortest distances and say which superstep now first reaches the correct answer for Zone Z.
   Solution

   Superstep 1 is unchanged (Kitchen=0, Hub-N=6, Hub-S=9, Hub-E=15, Zone-Z=∞). Superstep 2: Hub-E still revises to 11; Zone-Z now hears 9+3=12 from Hub-S (beating the old 17-via-Hub-N tie), so Zone-Z=12. Superstep 3: Hub-E's 11+3=14 arrives but 14>12, so nothing changes. The computation converges one superstep earlier than the original graph — in 3 supersteps rather than 4 — and the shortest path to Zone-Z now runs through Hub-S, not Hub-E.

3. Window 2's raw average (570.0s) undersells how slow deliveries actually were, because a wrongly-negative reading dragged it down. Could a wrongly-negative reading ever make a raw window average look artificially slow instead of artificially fast? Construct the condition under which that happens.
   Solution

   No single rule guarantees a direction — it depends on the sign and size of the corrupted value relative to the rest of the window. A large positive corrupted value (say a delivery mistakenly logged as 9,000 seconds instead of 900) would drag the raw average up, making deliveries look slower than they were. The lesson generalises past this one window: an unvalidated value doesn't bias an aggregate in a fixed direction, which is exactly why you check for validity rather than just "does the average look reasonable."

4. CityCourier wants both (a) total revenue per hub and (b) the shortest route from the Kitchen to every hub. Which of this section's models fits each task, and why can't MapReduce's two-phase shape comfortably do both?
   Solution

   (a) is embarrassingly parallel — every hub's revenue depends only on its own rows — so MapReduce's map-then-reduce shape fits directly, as Section 1.3's first worked example showed. (b) is inherently iterative: each hub's shortest distance depends on another hub's shortest distance, which is exactly the propagation pattern MapReduce has no native concept for. Forcing it into MapReduce would mean running a whole new MapReduce job per superstep, writing every intermediate result to disk between rounds — correct, but paying a full disk round-trip for what BSP does with one in-memory message pass. This gap is historically why Pregel-style, BSP-native systems were built at all.

Any large-scale system is layered, whatever else it does: raw events come in, get stored somewhere, get processed into something usable, and get served to whoever asked. The interesting design decisions live in exactly one question: how many processing paths does data take on its way from storage to a dashboard, and how do their answers get reconciled?

Worked Example · A Dashboard That Disagrees With Itself, On Purpose

Run CityCourier's revenue-per-hub figure (Section 1.3's MapReduce pattern, grouped by hub instead of restaurant) through both layers of a Lambda design and watch them disagree:

Neither number is a bug. The speed layer is answering "what do we know right now," honestly, from six fewer orders than exist. The serving layer's entire job is to display the speed layer's estimate live and quietly swap in the batch layer's exact figure once it lands — without that swap, "provisional" silently becomes "permanent," which is where dashboards start lying.

Lambda and Kappa, Side by Side

A Data Lake is neither of these paths — it's the raw, schema-on-read store the Event Log or Data Lake box above actually is: every field CityCourier ever collected, kept in its original shape, with structure applied only when something reads it. Both Lambda and Kappa can sit on top of one.

Practice

1. Using the two panels above, compute how much each hub's revenue figure will still move once the batch layer absorbs the remaining 6 orders.
   Solution

   Hub N: ₹2,085 − ₹1,375 = ₹710. Hub S: ₹1,695 − ₹765 = ₹930. Hub E: ₹1,765 − ₹1,150 = ₹615. These three differences sum to ₹2,255, which is exactly the revenue from orders #11–16 (520+165+330+410+545+285) — a useful sanity check that the batch/speed gap accounts for precisely the missing orders and nothing else.

2. CityCourier switches to a Kappa design and later discovers a bug: the streaming aggregator was mis-parsing amounts under ₹200. What does "replay to fix it" mean operationally, and what does Lambda's batch layer give you for free that Kappa now has to do deliberately?
   Solution

   Under Kappa, fixing the bug means deploying the corrected aggregation logic and re-running it over the entire retained event log from the start, producing a corrected history in one pass — the log itself never changes, only the logic reading it. Under Lambda, the batch layer already recomputes from raw data on every run by design, so a logic bug would have self-corrected at the very next scheduled batch job with no special "replay" step; Kappa gets the same correctness, but only if the team remembers to trigger it.

3. A shift manager glances at the dashboard at t = 422s, sees "Hub S: ₹765," and decides Hub S needs no extra staff today. What's the risk in that decision, given what you know about the serving layer?
   Solution

   ₹765 is the speed layer's figure after only 10 of 16 orders — Hub S's true total for just this lunch rush is ₹1,695, more than double. Deciding staffing from a number the serving layer itself labels as provisional risks under-resourcing the hub that, once the batch layer catches up, turns out to be busiest. The fix isn't to distrust the speed layer generally — it's to know which numbers on the dashboard are "as of now" and which are final.

4. Which of Section 1.3's distributed computing models most naturally implements a Lambda architecture's batch layer, and which implements its speed layer?
   Solution

   MapReduce (or a BSP-style iterative job, for anything that needs propagation) fits the batch layer: it assumes a finite, complete input and produces one exact answer per run, exactly matching the nightly full-log recompute. Stream processing fits the speed layer by definition — unbounded input, windowed, approximate, always live. Lambda is, in this sense, "pick one model from each side of Section 1.3's foundation and wire their outputs into one serving layer."

Four numbers, precisely defined, replace the word "faster":

Scalability is not a fifth number — it's what happens to Speedup and Efficiency as p grows. A perfectly scalable system has Speedup(p) = p (Efficiency = 100% at every p); every real system falls short of that line, and by how much is the whole engineering story.

Worked Example · Processing the 16-Order Batch

Give each order a fixed 40ms of aggregation work. One node does all 16 serially. Three nodes split the work along the hubs' natural partition (6 / 5 / 5 orders) and pay a fixed 55ms coordination cost once, at the end, to shuffle and merge — the same shuffle Section 1.3 already introduced.

Practice

1. Compute resource utilization for the 1-node case (p=1).
   Solution

   Utilization = useful work time ÷ (p × wall time) = 640ms ÷ (1 × 640ms) = 100%. With one node, "wall time" and "work time" are the same thing by definition — there is no possibility of idling relative to yourself, which is why p=1 is always the trivial 100% case.

2. If the shuffle overhead for the 3-node case rose from 55ms to 200ms (a slower network), is 3-node still faster than 1-node? At what overhead does 3-node stop being worth it at all?
   Solution

   At 200ms overhead: T₃ = 240 + 200 = 440ms, still comfortably below T₁'s 640ms — 3 nodes remain the better choice. The break-even point is where T₃ = T₁: 240 + X = 640, so X = 400ms. Below 400ms of shuffle overhead, splitting the work across 3 nodes wins; above it, the coordination cost eats the entire benefit and a single node would have finished first.

3. The 3-node split (72.3% efficient) beats the 2-node split (64.6% efficient) despite having more machines to coordinate. What does this imply about treating "scalability" as a property of node count alone?
   Solution

   It implies scalability is a property of the partitioning as much as the hardware. A system description that says "we scale to N nodes" without saying how work is divided among them is incomplete — the same N nodes can land anywhere from near-linear speedup to barely-better-than-serial, purely as a function of how evenly the load is split.

4. Section 1.3's BSP shortest-path example took 4 supersteps, each ending in a global barrier. If each barrier round-trip costs the same 55ms this section used for a single MapReduce shuffle, what's the total coordination overhead of the BSP example, and how does it compare to MapReduce's cost?
   Solution

   4 supersteps × 55ms ≈ 220ms of pure coordination overhead for BSP, versus one 55ms shuffle for MapReduce — roughly four times the coordination cost, purely because the problem needed four rounds of propagation rather than one. This is the concrete, measurable version of Section 1.3's claim that "more supersteps costs real wall-clock time": here is exactly how much, in the same units this section already established.

Part A · Consistency, Availability, Partition tolerance

A network partition is exactly what it sounds like: Hub S can no longer talk to the rest of CityCourier's cluster, but its own machine is fine and its own customers keep tapping "confirm." The CAP theorem, due to Eric Brewer, describes what a system can promise while that's happening:

Brewer's result is narrower than it's usually quoted: you cannot have all three while a partition is actually happening. Outside a partition — the overwhelming majority of the time — a well-built system gives you all three at once. The theorem is a statement about a failure mode, not a permanent tax on every request a distributed system ever serves.

Worked Example · Ninety Seconds Without Hub S

Orders arrive across the whole log at 16 orders in 636 seconds — about 0.0252 orders/second, or roughly 1.5 a minute. Suppose Hub S loses its link to the central cluster for 90 seconds during the rush. Expected orders arriving at Hub S in that window: 0.0252 × 90 ≈ 2. Small, but not zero — and what happens to those two orders is exactly the CP/AP choice.

Notice neither branch is "wrong." CityCourier choosing AP for order placement (a missed reconciliation is annoying but fixable) while choosing CP for, say, refund processing (a double refund is a real loss) is a completely reasonable, and common, design — CAP is a per-operation decision, not a single global setting for the whole system.

Part B · Storage, Computation, and Communication Cost

Every distributed operation spends three different kinds of resource: storage (keeping data somewhere), computation (doing something with it), and communication (moving it between machines that don't share memory). The general engineering principle — data locality, which Chapter 2's file systems are built around — is to minimise the most expensive of the three, and communication is very often it.

Worked Example · Ship the Data, or Ship the Question?

Each CityCourier order carries a GPS breadcrumb trail: 45 points × 24 bytes each = 1,080 bytes/order, × 16 orders = 17,280 bytes of raw tracking data. Someone at head office wants one number from it: each hub's average delivery distance. Two ways to get it, over the same 2 MB/s cross-town link:

bar width is proportional to bytes moved — the second bar is not a rendering error, it really is that thin.

Practice

1. Recompute the expected number of stranded orders if Hub S's partition lasted 240 seconds instead of 90.
   Solution

   0.0252 orders/s × 240s ≈ 6.04, so about 6 orders. The AP/CP trade-off scales directly with outage length: a 90-second gap risking 2 orders is a very different operational conversation from a 4-minute gap risking 6.

2. If the hub-to-central link were upgraded from 2 MB/s to 20 MB/s, recompute both transfer times. Does the 19× conclusion change?
   Solution

   Move-data time: 17,280 ÷ 20,000,000 = 0.864ms. Move-compute time: 924 ÷ 20,000,000 = 0.046ms. Both times shrink by exactly 10×, but their ratio is still 17,280/924 ≈ 18.7× — unchanged, because bandwidth divides both sides of the ratio equally. The lesson: the "ship compute, not data" conclusion here is about the shape of the two payloads, not about how fast any particular link happens to be.

3. CityCourier later finds that one of the two orders Hub S accepted locally during the outage shares an order ID with one placed through the main app during the same window. What class of problem is this, and roughly how is it normally resolved?
   Solution

   This is a write conflict created by choosing Availability during a partition — two writes to the same logical record, accepted independently, that now disagree. It is not data loss and not a bug in either hub; it's the expected cost of an AP choice. Resolution typically means detecting the conflict (comparing timestamps, version numbers, or a "last write wins" / application-specific merge rule) and applying a reconciliation rule once the partition heals — the mechanics of exactly how belong to Chapter 2's replication strategies.

4. If Hub S chooses Availability during the partition — serving customers from its own local state without waiting for the central cluster — is it also, implicitly, choosing to move computation to data rather than data to computation? Explain the connection.
   Solution

   Yes. Choosing Availability requires Hub S to make a decision using only what it has locally, which is only possible if the logic needed to decide (validate an order, compute a price) can run at the hub itself — exactly the "ship the question, not the answer" pattern from Part B. A CP design that insists on checking with the central cluster before responding is, structurally, choosing to ship the request to where the authoritative data lives instead. The CAP choice and the cost-locality choice are two views of the same underlying decision: where does the deciding actually happen?

Every earlier section in this chapter treated the CityCourier log as an engineering object: something to count, route through, architect around, and time. Governance asks a different question of the exact same sixteen rows — not can you compute an answer, but should you act on it, and who answers for it if it turns out to be wrong.

Worked Example · The Ranking That Wasn't Real

Recall order #10 from Section 1.1: Hub E, delivery time logged as −60 seconds. Compute each hub's average delivery time — a completely reasonable thing for a performance dashboard to show — twice: once trusting every value as logged, once applying the same veracity filter from Section 1.1.

One corrupted reading — a courier's clock, not even a customer's private data — hands Hub E first place on the raw dashboard. Clean the input and Hub E drops to a distant second; Hub N, never mentioned in the "fastest hub" conversation, was actually fastest all along. If a bonus, a staffing decision, or a public "our fastest hub" press line got attached to the raw number, Section 1.1's veracity bug has quietly become a fairness problem: Hub N's team did the better work and Hub E's team got the credit for it, because nobody checked whether −60 seconds was possible before an average got computed from it.

Practice

1. Using the veracity-filtered table, rank the three hubs from fastest to slowest average delivery.
   Solution

   Hub N (682.0s) fastest, Hub E (716.25s) second, Hub S (738.0s) slowest.

2. CityCourier's policy is "the slowest hub each week loses its bonus." Compare which hub loses the bonus under the raw numbers versus the veracity-filtered numbers. Which of the two rankings — "fastest" or "slowest" — actually changed once the data was cleaned, and what does that tell you about when a data-quality bug matters most?
   Solution

   Hub S is slowest under both the raw (738.0) and filtered (738.0) numbers — the bonus-loss decision doesn't change. What changes is who is credited as fastest: Hub E under raw data, Hub N once cleaned. The lesson is specific, not "always recheck everything equally": a data-quality bug matters most exactly where the corrupted record sits — it inflated Hub E's own number, so it distorts comparisons Hub E is party to, and leaves comparisons among the other hubs untouched.

3. Suppose CityCourier automatically flags any courier whose average delivery time exceeds 800 seconds for review. Explain how the same −60-second bug that wrongly crowned Hub E "fastest" could, in a different pipeline, just as easily let a genuinely underperforming courier avoid being flagged at all.
   Solution

   A large negative value pulls an average down. If that same courier had several legitimately slow deliveries pushing their average toward the 800s threshold, one corrupted −60 reading averaged in alongside them could drag the mean back under 800 and suppress the flag — the identical mechanism that inflated Hub E's ranking here would, for an individual courier, hide a real problem instead of manufacturing a fake one. Section 1.3 made the general point that bad data doesn't bias an aggregate in a fixed direction; this is that same point with a governance consequence attached.

4. Which earlier section's concept is the root cause of this section's ranking flip, and at which of Section 1.2's life-cycle stages should it have been caught?
   Solution

   Section 1.1's Veracity problem — order #10's physically impossible delivery time — is the entire root cause; nothing in this section introduced a new error, it only showed a consequence of the old one. Per Section 1.2, it should have been caught at Ingest, the last point where a single bad value can be rejected before it is durably stored, replicated, and eventually folded into an aggregate that a real staffing or bonus decision relies on.

1. A distributed store keeps serving reads and writes through a network failure, but two replicas briefly disagree once the failure clears. Which side of the CP/AP choice did it make?
   Solution

   Availability. It kept answering requests through the partition (never refused or hung) at the cost of a brief consistency gap between replicas — the AP branch of the trade-off in Section 1.6.

2. A batch job takes 800ms on one node and 250ms on four. Compute its speedup and efficiency at p=4.
   Solution

   Speedup = 800/250 = 3.2×. Efficiency = 3.2/4 = 80%.

3. A delivery record shows a 5-star rating and a 3-second delivery time for a 40-minute cross-town trip. Is this missing data or wrong data, and which characteristic does it violate?
   Solution

   Wrong data — every field is populated, but the delivery time is physically implausible. This is a Veracity violation, the same category as CityCourier's order #10.

4. Name the two processing layers in a Lambda architecture, and say which one a live dashboard figure is usually reading from first.
   Solution

   The batch layer (slow, exact) and the speed layer (fast, approximate). A live dashboard reads the speed layer first; the serving layer swaps in the batch layer's corrected number once it's ready.

5. In a BSP computation, a vertex receives no message for two consecutive supersteps. What does it do, and what would make it active again?
   Solution

   It votes to halt and stays idle. It reactivates only if it receives a new message from a neighbour in some later superstep.

6. A company strips customer names from a GPS tracking dataset before sharing it with a partner. Is that sufficient for privacy?
   Solution

   Not necessarily. A repeated location pattern (the same address every evening) can re-identify a person without a name field at all — privacy has to account for what combinations of fields reveal, not just which fields are labelled personal.

7. Grouping a thousand (word, count) pairs so every count for the same word reaches the same reducer is which phase of MapReduce?
   Solution

   Shuffle (with sort) — the one phase that crosses the network, between the local map phase and the local reduce phase.

8. A system ships a 50KB scoring model out to where a 50GB dataset lives, instead of shipping the data to a central server. What principle is this, and under what condition would the right choice reverse?
   Solution

   Moving computation to data (data locality), to minimise communication cost when the data is far larger than the code. It reverses if the computation itself becomes the larger payload — a huge model scoring a tiny dataset is cheaper to run by shipping the small data to the model, not the other way around.

• Mining of Massive Datasets — Leskovec, Rajaraman & Ullman. The MapReduce and stream-mining chapters go well beyond word count; good next stop after Section 1.3.
• Designing Data-Intensive Applications — Martin Kleppmann. The single best full-length treatment of the CAP theorem and replication trade-offs behind Section 1.6; worth reading before Chapter 2.
• Streaming Systems — Akidau, Chernyak & Lax. Goes deep on exactly the event-time-vs-processing-time questions Section 1.3's tumbling window only touches; essential before Chapter 5.