Nobody tells you the answer — or the question.

The three answers differ by an order of magnitude on identical rewards, and the ordering of what matters flips: at γ = 0.5 the near +2 contributes 0.5 and the distant +5 only 0.156, while at γ = 0.9 the +5 dominates at 2.95. γ is not a tuning knob for convergence speed; it changes which policy is optimal.

The ranking is identical to Worked 6.1's — (a) then (b) then (c) — so this formulation also prefers the short safe path, and either design would train a correct agent. The numbers differ because they answer slightly different questions: discounting says "later is worth less", while a step penalty says "moving costs something".

The practical difference appears when episodes can fail to terminate. With γ = 1 and no step penalty, an agent that circles forever accumulates 0 and is not punished; with a step penalty it accumulates −∞ and is strongly punished. So a step penalty makes γ = 1 safe on tasks that might loop, which is exactly why the classic textbook grid world uses −0.04.

The agent is doing exactly what it was paid for, and the reward function is an incomplete statement of the job. Nothing in +1 per parcel mentions aisles, so obstructing one is free; nothing distinguishes an easy parcel from a hard one, so ignoring high shelves is optimal if they take longer. Both behaviours increase the reward. This is not a training failure and more training will make it worse.

What to change. Add the missing costs explicitly — a penalty for time spent stationary in an aisle, and a time cost per delivery so that a slow parcel is worse than a fast one rather than equally good. Consider rewarding completion of an assigned manifest rather than parcels freely chosen, which removes the incentive to cherry-pick. And check whether the shelf problem is really a reward issue at all: if the robot physically cannot reach high shelves, no reward fixes it and the diagnosis is mechanical.

What not to do is add a large penalty for blocking aisles and stop. That invites the next unintended solution — refusing to enter aisles — and the pattern repeats. The reliable move is to write down what a good outcome actually is, in full, and to watch the resulting behaviour rather than only the score. Which is 4.8's reporting argument in a setting where the model's output is behaviour rather than a prediction.

Supervised learning has an external referent. Section 2.1's protocol works because held-out labels exist independently of the model, so "is this prediction right" is a well-posed question with an answer nobody had to invent.

Unsupervised learning has none, which was section 3.5's central difficulty. "Better" had to be assembled from geometry, stability under resampling, and comparison against a null model — three weaker substitutes.

Reinforcement learning has a referent, but you designed it. Reward is an objective, measurable signal, so unlike clustering there is a definite sense in which one policy beats another. But the signal is your own specification rather than a fact about the world, so a high score means the agent satisfied your statement of the goal, which the depth box shows is a different claim from satisfying the goal.

And there is a difficulty neither of the others has: no held-out set. The agent generates its own data by acting, so it cannot observe the counterfactual and cannot be evaluated on experience it did not have. Section 6.4's answer is to report several different quantities — cumulative reward, regret, episode length, success rate — because each is gameable alone, and to average over many seeds because the variance between runs is larger here than anywhere else in the course.

The optimal policy is exactly the same. Every value is 0.5 raised to the steps remaining rather than 0.9, so the ordering of actions in every state is unchanged and the same two shortest paths are optimal.

That is the general situation when there is a single positive terminal reward and no step cost: γ rescales the values but cannot reorder them, because a shorter path is better at every γ < 1. γ changes the policy only when there is a genuine trade between a small near reward and a large distant one — which is exactly the local-optimum grid of section 6.3, where at low γ the near +0.3 would genuinely be worth more.

Still clearly the worst action at (0,1), but noticeably less bad than the deterministic −1.0000, because a fifth of the time the slip saves the agent from its own decision.

Two honest caveats. The true slippery values differ from the deterministic ones, so a proper solution requires re-running value iteration on the stochastic MDP rather than substituting as done here — this calculation is illustrative of the form, not exact. And slipperiness changes the optimal policy in a specific direction: it makes the agent prefer routes that stay away from the pit even at the cost of extra steps, because a slip near a hazard is expensive. That is the standard lesson of the classic slippery grid world.

It almost certainly fails, and the failures are enumerable. The number of free beds does not determine the distribution of what happens next, because several things that matter are missing from it. Time: admissions have a strong daily and weekly cycle, so five free beds at 3 a.m. on a Tuesday and five on a Friday evening lead to very different futures. Who is in the beds: ten patients due for discharge tomorrow is a different state from ten admitted this morning, and the count cannot distinguish them. Pending arrivals: an ambulance already en route is information about the next transition that the count omits.

The fix is to enlarge the state until the property holds, which is a modelling decision, not a learning one. Add time of day and day of week, a summary of expected discharges over the next day, current admissions in progress, and perhaps a coarse case-mix. Each addition makes the assumption more nearly true and makes the state space larger, which is the trade the depth of section 6.3 will describe.

And check whether the enlarged space is still tractable. Free beds (0–50) × hour (24) × day (7) × expected discharges (0–20) is already about 176,000 states, which a table can just about hold but which will need a great deal of data to fill. Beyond that, tabular methods stop being available and function approximation is required, which is where Unit 5's networks re-enter the story.

Worth adding that a violated Markov assumption does not announce itself. The algorithm will run, converge to something, and be systematically wrong in ways that look like noise — so the assumption should be argued for before training, not diagnosed after.

The alternating structure is familiar. Value iteration alternates between evaluating the current estimate and improving it, exactly as k-means alternates assign and update (3.1.1) and as EM alternates E and M steps (3.1.4). All three are fixed-point iterations that improve a quantity monotonically until nothing changes, and all three converge because each step cannot make things worse.

What is genuinely different is the guarantee. k-means and EM converge to a local optimum, and section 3.1.1 measured the cost: 28 of 220 initialisations reached a worse answer, silently. Value iteration on a finite MDP with γ < 1 converges to the global optimum from any starting point, because the Bellman operator is a contraction with modulus γ — each sweep shrinks the error by a factor of at least γ. There are no local optima to be trapped in, and no restarts are needed. That is a much stronger guarantee than anything in Units 3 to 5.

What it costs is knowledge of the environment. Backward induction needs P and R in advance — a complete map. k-means needed only the data. So the strong guarantee is bought with an assumption that is usually unavailable, and section 6.3 gives it up: Q-learning reaches the same Q* using only sampled experience, and the price is that convergence becomes asymptotic and depends on visiting every state-action pair often enough.

The connection to 1.4 is the update itself. Value iteration's sweep and Q-learning's rule both move a current estimate toward a target, and section 6.3 will show that Q ← Q + α[target − Q] has precisely the shape of a gradient-descent step with α as the learning rate — the same object, in a setting where the target is itself an estimate.

The two answers differ by 0.315 on identical r, α and γ, purely because of whether the next state is terminal. Note the direction in the terminal case: because the episode ended with no reward, the estimate is revised downward — the action led nowhere and the update says so.

Worked 6.3a's first three values — 0.5000, 0.7500, 0.8750 — are exactly this sequence, which is a useful check on that trace. The convergence is geometric with ratio 1−α, so a larger α is strictly faster when the target is constant.

The catch is that the target is usually not constant. In a stochastic environment each visit gives a different sample, and α = 0.5 then means the estimate tracks the last few samples rather than averaging over all of them — so it converges fast to something that keeps jittering. That is why the theory requires α to decay, and why a fixed α is a practical compromise rather than a correct choice.

The plateau has two candidate causes and they need opposite fixes, so identify which before acting.

Cause 1: ε = 0.3 is the ceiling, not the policy. An agent that acts randomly 30% of the time cannot achieve the optimal return even with a perfect Q-table, because three moves in ten are thrown away. Worked 6.3b measured exactly this: at ε = 0.30 the mean return was 0.6502 against an optimal 0.8100, with a perfect policy underneath. Test it by evaluating greedily — run episodes with ε = 0 and no learning. If the greedy return is near optimal, nothing is wrong with the agent; the reported number was measuring the exploration premium, and the fix is to decay ε and to report greedy performance separately from training performance.

Cause 2: it is genuinely stuck. If greedy evaluation is also poor, then the plateau is real. Then look for a local optimum of the kind Experiment 2 constructed — a modest reward that is easy to reach and self-confirming. Check whether any state-action pairs remain unvisited after 10,000 episodes, which points at insufficient exploration in a specific region rather than in general. Check the learning rate: a fixed α that is too small learns too slowly to escape, too large and the estimates never settle.

And check the reward function, per 6.1's depth box, because a plateau below what you consider optimal sometimes means the agent has correctly optimised a specification that does not match your intention.

The form is gradient descent's. Section 1.4's update was w ← w − η∇L — move the current estimate in proportion to an error signal. Q-learning's is Q ← Q + αδ, with δ the temporal-difference error and α the learning rate. Section 4.5's stability analysis transfers directly in spirit: too large and the estimate oscillates, too small and it never arrives, and the optimisers of that section are what deep Q-learning actually uses on θ.

What is new is that the target is an estimate. Gradient descent computes an error against a fixed known label. Q-learning computes it against r + γ max Q(s',a'), which contains Q — the thing being learned. That is bootstrapping, and it is why the convergence proof is delicate, why a target network is needed once Q is a network, and why Worked 6.3a's reward propagates only one step per episode: each square can only learn once its neighbour has.

The alternating structure is 3.1.1's. Evaluate the current estimate, improve the policy implied by it, repeat — the same shape as k-means' assign-and-update and EM's E-and-M. And the exploration dilemma is 3.1.1's initialisation problem with a twist: k-means solved a poor local optimum by restarting, which a reinforcement-learning agent cannot do, because a restart does not undo the reward already forgone. Exploration is the only tool available, and unlike a restart it has to be paid for out of the same budget being optimised.

Which is the genuinely new thing in the whole unit. Every earlier method could look at data for free. Here, gathering information and performing well are in direct competition, and Worked 6.3b's two experiments show the trade going both ways depending on the environment. No amount of algorithmic cleverness removes it; it is a property of learning by acting.

The two numbers answer different questions and both are worth reporting. Total regret 2.07 says learning cost the equivalent of about 2.5 optimal episodes — the price of getting here. Mean return 0.7333 describes recent behaviour but is dragged down by episodes 4 and 5, which are already history; the final episode alone achieved the optimum. On a short run like this, quoting the last episode is over-optimistic and the last-three mean is pessimistic, which is why a learning curve is better than either.

If learning happens offline, in simulation, before deployment — prefer neither, or prefer B if it is more robust. Regret accumulated in a simulator costs compute, not consequences. If both reach the same policy, 340 units of simulated regret is a wall-clock cost and nothing more, and A's advantage largely evaporates.

If learning happens online, on the real system, prefer A decisively. Regret is then real: real recommendations shown to real users, real trades placed, real medication doses given. Agent B was wrong 28 times as expensively on the way to the same answer.

And there is a case for B that the numbers hide. Low regret means the agent explored little, which means it committed early to what it found. On this environment that worked, but section 6.3's second experiment showed a greedy agent locking onto a local optimum and reaching the real goal 1.5% of the time. B's higher regret bought coverage, and coverage is insurance against an environment slightly different from the one tested. If the deployment environment may differ from the training one, B's exploration is a feature.

So the honest answer is that regret alone cannot decide it: you need to know whether the regret was paid in simulation or in reality, and whether the environment is fully known.

Almost everything needed to evaluate the claim, and this is the standard failure the depth box describes.

No seeds and no spread. A single run of each, on a difference of 9%. Reinforcement-learning variance between seeds routinely exceeds that, so the result may reverse on a re-run. Report at least 5 to 10 seeds with a mean and interquartile range, and compare properly rather than by eye.

No scale. 4,200 of what maximum? Without the optimal value, or at minimum a random-policy baseline and a human score, the number is uninterpretable — this is 2.4.2's majority-class argument, unchanged.

No statement of training or evaluation. Was 4,200 earned while exploring, or by a greedy evaluation with learning off? The two differ substantially and only the second describes the method.

No budget and no environment version. A method that wins with ten times the samples has not won on equal terms, so report sample counts and wall-clock cost. And reward scaling or termination rules differ between environment versions, so the version is part of the result.

No learning curve. The final number cannot distinguish an agent that converged quickly and sat at 4,200 from one that spiked there on the last evaluation. The curve, with a spread band across seeds, is the honest presentation and would show both.

Supervised learning had an external referent and a protocol for using it. Section 2.4.6's rule — select on validation, estimate on test, report a spread — works because held-out labels exist independently of the model. The metrics could be gamed, but the ground truth could not.

Unsupervised learning had no referent at all, which was section 3.5's difficulty. "Better" had to be assembled from geometry, stability and a null baseline, and the honest conclusion was that unsupervised model selection is fundamentally weaker.

Reinforcement learning has a referent that you wrote. Reward is objective and measurable, so one policy definitely beats another under it — a real advantage over Unit 3. But the referent is a specification rather than a fact, so maximising it is not the same as succeeding, and 6.1's reward-hacking examples are what that gap looks like when an optimiser finds it. In a precise sense this is the sharpest version of a problem the course has raised repeatedly: 2.4.2's accuracy, 3.2's silhouette and 4.8's calibration were all proxies too, but here the proxy determines behaviour rather than a prediction, so the consequences are actions in the world.

And one difficulty is entirely new. Every earlier setting could evaluate on data the model did not influence. Here the agent generates its own data, so there is no held-out experience, no counterfactual, and no way to ask what would have happened under a different policy without running it. Section 4.8's insistence on several seeds becomes a requirement rather than good practice, and the separation of training from greedy evaluation is the only substitute available for a train/test split.

The thread running through all four units is a single question: what is this number entitled to claim? The answer has become progressively more restrictive — from "this model predicts held-out labels this well", to "this partition is geometrically tidy and reproducible", to "this agent maximised the objective I wrote, in the environment I tested, from these seeds". Learning to state the last one precisely is most of what a machine learning education is for.

At γ = 0.8 the distant +3 still dominates, because 0.8³ = 0.512 retains over half its value. Drop to γ = 0.5 and it contributes 0.125(3) = 0.375 against the immediate 1, so the ordering of what matters flips. Whether a distant reward is worth pursuing is set by γ, not by the agent.

Note the two wall moves are worth 0.81 — more than moving up to a genuinely worse square. Bouncing off a wall wastes one step and costs a single factor of γ; moving up costs a step and lands somewhere further from the goal. And the check that maxₐ Q*(s,a) = V*(s) holds is worth doing every time, since it catches most arithmetic slips.

The tie matters more than it looks. If the implementation always breaks ties by taking the first index, action 2 gets 0.85 and action 4 gets 0.05, so the agent systematically ignores an equally good action — and on the spine grid, where the start square has exactly this tie, that means one of the two optimal paths is never explored. Random tie-breaking is a real design decision, not tidiness, and section 6.3 showed it is the mechanism by which a greedy agent explores at all.

The closed form is the same geometric decay as P6.3.2, with the target at −1 instead of +1: the error shrinks by a factor of 1−α each visit and the estimate approaches the target from wherever it started. Note that one bad experience is not enough to make the action look bad here — the estimate is still positive after the first update — which is why a single catastrophic outcome may be repeated before it is learned.

Off-policy means the policy being learned about is not the policy generating the behaviour. The update's target is r + γ maxₐ' Q(s',a'), which asks "what is this worth if I play optimally from here", regardless of what the agent will actually do next. So the agent can behave randomly, or follow a fixed demonstration, or replay old experience from a policy it no longer uses, and still converge on Q*.

That property is what makes exploration safe. An ε-greedy agent takes deliberately bad actions and its Q-table is not corrupted by them, because the max in the target ignores the exploration. It is also what makes experience replay possible, since stored transitions come from an older, worse policy.

Replacing the max with the action actually taken gives SARSA, an on-policy method, and it learns Qᴵ for the exploring policy rather than Q*. The practical difference is instructive: on a grid with a hazard beside the optimal path, SARSA learns that walking beside the pit is dangerous because it sometimes explores into it, and so prefers a longer safer route. Q-learning learns the optimal path and then walks it while occasionally falling in. Neither is wrong — SARSA evaluates the policy you will actually run, which is the right thing when exploration cannot be switched off in deployment.

The memory is merely awkward; the visits are the fatal problem. Even at a thousand transitions per second, visiting each entry once takes about nine days, and Q-learning needs many visits per entry. Worse, the table cannot generalise at all: having learned about one joint configuration it knows nothing about the configuration one increment away, even though they are nearly identical physically.

So this is exactly where the table must be replaced by function approximation. Represent Q(s,a;θ) as a network taking the six joint angles as input, and a single update improves the estimate for every similar configuration at once. That generalisation is the point, not the memory saving — and it is why Units 4 and 5 are prerequisites for anything beyond a toy problem, even though this unit needed none of them.

The conclusion does not follow, because the 0.68 was measured while the agent was deliberately acting randomly a fifth of the time. Section 6.4's distinction applies directly: this is a training number, and it includes the exploration premium. The policy underneath may be exactly optimal.

The test is one line: evaluate greedily. Run episodes with ε = 0 and learning switched off. Worked 6.3b measured precisely this situation — at ε = 0.30 the training return was 0.6502 while the underlying policy was very nearly optimal — so the expected outcome here is a greedy return close to 0.81 and a conclusion of "the policy is fine; the report was measuring the wrong thing".

If greedy evaluation also gives 0.68, then the criticism stands and the diagnosis moves on: check for unvisited state-action pairs, check for a local optimum of the kind section 6.3's second experiment constructed, and check the learning rate. But that is a different investigation and it should not be started until the cheap test has been run.

And in either case the report is inadequate. One number, no seeds, no spread, and no statement of whether it was a training or an evaluation figure. Section 6.4's checklist asks for all four measures with their optimal values beside them, over at least five seeds.

It will learn to maximise clicks, which is not the same as being useful, and the gap is exploitable in specific ways. Expect it to favour content that provokes rather than informs, since outrage and curiosity gaps are reliably clickable. Expect clickbait framing over accurate framing. Expect it to recommend items the user would have found anyway, because those click reliably. And expect no weight at all on whether the user was glad afterwards, because nothing in the reward mentions it.

This is 6.1's reward-hacking argument, and it is not hypothetical — engagement-optimised recommendation is the most consequential deployed example of it.

A better reward has to state more of what you want. Weight a click by dwell time or completion, so a click followed by an immediate bounce is worth little. Include an explicit negative for a hide, a report or an unfollow. Reward returning tomorrow rather than clicking today, which converts a one-step objective into a long-horizon one and is where the discount factor earns its place. And add a diversity term, so that exploiting one narrow interest is penalised.

Two cautions, though. Every added term is another surface to hack, so the list above is an improvement rather than a solution. And some of what you want — whether the recommendation was good for the person — may not be measurable from behaviour at all, in which case no reward function captures it and the honest response is to constrain the agent's action space rather than to keep refining the reward. Section 2.6's argument, arriving where the model's output is behaviour.

The greedy policy is optimal, and the agent has still not learned the environment. Its error is on an action it never takes, so the error is invisible in its behaviour and costs nothing while conditions hold. This is a good illustration of why Q-learning converges only under the condition that every state-action pair is visited infinitely often: without it, Q ≠ Q*, even when π = π*.

The risk is that the policy is correct for the wrong reason and is not robust. If anything perturbs the values — a change in the reward scale, a stochastic transition, a small negative step cost that pushes Q(down) below 0 — then Q(right) = 0 becomes the maximum and the agent walks confidently into the pit. An unexplored action with an optimistic default is a latent failure, and zero-initialisation is exactly such a default when the true value is negative.

Which is the argument for exploration that section 6.3's first experiment appeared to undermine. On that plain grid, greedy was optimal and cheapest; this question shows the sense in which it was nonetheless underinformed, and the trap experiment shows what happens when that underinformation matters.

Supervised learning requires labelled examples and provides a function that predicts them. Its great advantage is an external referent: held-out labels exist independently of the model, so "is this prediction correct" has an answer nobody invented. Units 1, 2, 4 and 5 are all this, with progressively more flexible function classes — a line, then six model families, then a network, then a network with architectural assumptions baked in.

Unsupervised learning requires only data and provides a summary of its structure. It has no external referent, which was section 3.5's difficulty: "better" had to be assembled from geometry, reproducibility and a null baseline, and the honest conclusion was that its model selection is fundamentally weaker. Unit 3's payoff figure — PCA optimally retaining 56.7% of the variance while collapsing two of three clusters — is the sharpest statement of what a proxy objective can cost.

Reinforcement learning requires an environment it can act in and provides a policy. Its referent is objective but self-authored: reward is measurable, so one policy definitely beats another, yet a high score certifies only that your specification was maximised. And it is the only one of the three where gathering information competes directly with performing well, so exploration must be paid for out of the budget being optimised.

The question running through all three is what a number is entitled to claim, and the answer has narrowed at every step. Unit 2 could say "this model predicts held-out labels this well, with this spread, on data used once". Unit 3 could say only "this partition is geometrically tidy and survives resampling". Unit 6 can say "this agent maximised the objective I wrote, in the environment I tested, from these seeds, and here is the exploration premium separated out".

Every unit produced at least one number that meant less than it appeared to: accuracy holding at 0.7000 across three genuinely different classifiers; every internal index preferring a finer clustering than the truth; a 56.7%-variance projection destroying the structure; three AUCs of exactly 1.0000 beside an accuracy of 0.8333; and a training return of 0.6502 concealing a near-optimal policy. Learning to state precisely what a result does and does not establish is not a supplement to the technical material. It is the technical material.