Four points that break everything so far.

OR converges faster than AND (4 updates against 10) because its positive region is larger, so a randomly placed boundary is more likely to be nearly right. Verify the answer: x₁ + x₂ > 0 fires for B, C and D and not for A. ✓

NAND is A→1, B→1, C→1, D→0 — the negation of AND. Negate AND's solution from Worked 4.1a: w = (−2, −1), b = +2.

C fails on the strict convention, so shift the bias slightly: w = (−2, −1), b = +2.5 gives +0.5 → 1 for C and −0.5 → 0 for D. ✓ The general recipe is that negating a separable function is always separable — flip w and b and adjust for the boundary convention.

Why no contradiction. Separability is a property of a specific labelling of specific points, not of "logic functions" as a class. Of the sixteen Boolean functions of two variables, fourteen are linearly separable; only XOR and XNOR are not. What makes XOR special is that its two positive examples sit on opposite corners of the square, so any line separating them from the other pair would have to bend.

The historically important footnote: NAND is functionally complete, so a network of NAND gates can compute anything, including XOR. Representational power comes from composition, which is the argument for depth and precisely what the perceptron lacked.

With 200 features and 150 examples, a linear model can almost always achieve 100% training accuracy on any labelling, so the result carries essentially no information. Points in general position in d dimensions are linearly separable whenever n ≤ d + 1; here n = 150 and d = 200, so separability is close to guaranteed regardless of whether the labels mean anything. Shuffle the labels at random and the same 100% would appear.

This is 1.6's bias-variance material and 2.1's protocol arriving together. Training accuracy measures capacity, not fit, and here the capacity exceeds the data by a wide margin.

What would settle it. The label-permutation test, which is the decisive one: retrain on shuffled labels, and if training accuracy is still 100%, the original result was capacity. A proper validation split or cross-validation, per 2.1. Regularization — ridge or a strong weight penalty — which will collapse a spurious fit and leave a genuine one standing. And more data, since n < d is the root problem.

The hypothesis space is identical. All three output sign(wᵀx + b) and can represent exactly the linearly separable functions, so all three fail on XOR for the same reason. What differs is entirely the objective.

The perceptron minimises nothing. It has an error-driven update rule with a convergence proof, not a loss function it descends. Consequently it stops at the first separator it finds, and which one that is depends on the data order and the initialisation. On non-separable data it does not terminate.

Logistic regression minimises cross-entropy, a smooth convex loss, so it has a unique optimum (with regularization), it converges on non-separable data, and it produces calibrated probabilities rather than bare decisions. That last property is what made all of 2.4.3 to 2.4.6 possible.

The SVM minimises hinge loss plus a norm penalty, which selects the maximum-margin separator specifically. Worked 4.1a's answer put the boundary exactly through point C, at zero margin; the SVM exists to rule that out. It also gains the kernel trick, which is a different route to non-linearity — a fixed, hand-chosen feature map instead of a learned one.

The relationship to this unit. A network with no hidden layer, one sigmoid output and cross-entropy loss is logistic regression, trained by gradient descent rather than by a dedicated solver. So this unit does not replace 2.3.1; it contains it as the depth-zero case, and everything new comes from what the hidden layers add.

Case (d) is AlexNet's first layer and shows the floor doing real work: 213/4 = 53.25, so the final window overhangs the edge and is dropped. Case (c) is the standard "halve the resolution" block — padding 1 with a 3×3 kernel keeps the arithmetic clean so that stride 2 divides exactly.

The blur kernel averages the nine values in each window, so each output is the count of 1s in the window divided by 9.

Every output is positive and they vary only slightly — between 0.5556 and 0.6667 across these four. The blur kernel's weights sum to 1 rather than 0, so it responds to average brightness rather than to change, and on a roughly uniform image it produces a roughly uniform map.

That is the general rule worth extracting: a kernel whose weights sum to zero is an edge or change detector; a kernel whose weights sum to one is a smoother. Trained networks discover both kinds without being told, and the first convolutional layer of a trained vision model is usually recognisably full of edge detectors at various orientations plus a few colour blobs.

Count the parameters first, because the diagnosis is in the arithmetic. After one conv and one pool the feature map is 100×100×32, which flattens to 320,000. The dense layer then costs (320,000 + 1) × 512 ≈ 163.8 million parameters, against 320 in the convolutional layer. Over 99.9% of the network is a single dense layer, and medical image datasets are rarely larger than a few thousand images.

So this is 1.6's variance problem in its purest form: a 23-point gap between training and validation, produced by a model with more parameters than it could possibly constrain. It is barely a convolutional network at all — it is a dense network with one convolution glued to the front.

The fixes, in order of impact. Replace flatten with global average pooling, which turns the 100×100×32 map into a 32-vector and cuts that layer from 163.8 million to about 16,900. Add more conv-and-pool stages before flattening, so the spatial dimensions come down properly — four more pooling stages would reduce 100×100 to about 6×6. Then add the standard regularisers of section 4.6 and data augmentation, which for medical images usually means flips and small rotations. And check for leakage: if multiple slices from the same patient appear in both splits, the 71% is optimistic too, and grouped splitting is required.

A convolutional layer is a dense layer with most weights forced to zero and the rest tied together. Write the convolution as a matrix mapping all input pixels to all output positions. The matrix is enormous, but it has two special properties: each output row is non-zero only at the k² pixels inside its window (sparse connectivity), and every row contains the same k² values in shifted positions (weight tying). A dense layer is the same matrix with those constraints removed.

Both constraints are regularization, in exactly 1.7's sense: they restrict the hypothesis space to reduce variance at the cost of some bias. The difference from ridge or lasso is that these are hard constraints derived from a fact about the data — that images have local structure and that patterns mean the same thing wherever they appear — rather than soft penalties tuned by cross-validation. A hard constraint that is true is the most efficient regulariser available, because it costs no bias at all.

And when the assumption is false, the constraint is pure bias. On tabular data where column 5 and column 6 have no relationship, convolution's locality assumption is simply wrong and a dense layer will do better. That is the same trade as everywhere in this course: 3.1.1's Voronoi cells, 2.3.4's axis-aligned splits, and a convolution's locality are all restrictions on shape, useful exactly when they match the data and harmful when they do not.

The connection to 4.2's parameter count now reads differently. The convolutional layers were cheap not because convolution is a clever trick, but because a true assumption let the network stop paying for flexibility it did not need.

The derivatives of sigmoid and tanh are symmetric about zero, so saturation is equally bad in both directions. ReLU's are not symmetric at all, which is exactly the asymmetry that makes it both immune to positive-side saturation and vulnerable to dying.

In 32-bit floating point, numbers near 10⁻¹⁴ are still representable, so this does not underflow to zero — it is worse than that in practice, because the gradient is nonzero but utterly swamped by the later layers' updates. The first layer effectively never moves, so the network behaves like a 2- or 3-layer model with a random fixed front end.

What to change, in order. Replace the activations with ReLU or leaky ReLU, which removes the factor entirely on the positive side — this alone fixes it. Add residual connections, Unit 5's mechanism, which give the gradient an additive path that skips the activation. Add batch normalisation (4.6), which keeps pre-activations near zero where σ' is largest, though even the best case of 0.25¹⁴ = 3.7 × 10⁻⁹ is still fatal, so this helps without solving it. And check initialisation (4.7), since |z| = 2 at every layer suggests weights that are too large.

The historical point: this calculation is why deep networks did not work before about 2010. Nothing about the architecture was wrong; the activation function was.

Distinguish two things that look identical in that statistic. A ReLU network is supposed to have sparse activations — on any given input, a large fraction of units being zero is normal and is part of why ReLU works. The question is whether the same units are zero for every input.

The diagnostic. Compute, for each unit, the fraction of examples on which it is non-zero. If most units fire on 40–70% of inputs and few are always zero, the network is healthy. If 40% of units have a firing rate of exactly 0.000, those units are dead: they contribute nothing to the forward pass and receive no gradient, so the network's effective width is 60% of what you are paying for.

If they are genuinely dead, the cause is almost always the learning rate. A single oversized update pushes a unit's pre-activation negative everywhere and there is no way back. Check whether the death happened early in training, which the diagnostic will show if you log it per epoch. Reduce the learning rate, add a warmup, or switch to leaky ReLU, which caps the damage because the gradient never reaches exactly zero. Also check the biases: large negative bias initialisation causes this directly.

And a note on severity. Dead units waste capacity but do not corrupt what remains, so a network with 40% dead units still works — it just works like a smaller network that you trained expensively. The reason to fix it is efficiency, not correctness.

On the output layer, saturation is cancelled by the loss. Section 4.1 showed that pairing a sigmoid output with cross-entropy gives ∂L/∂z = ŷ − y — the sigmoid's derivative divides out exactly. So a saturated output unit that is confidently wrong still produces a gradient of magnitude near 1. The pathology that makes sigmoid unusable in hidden layers does not arise, because nothing multiplies by σ' there.

In a hidden layer nothing cancels it. The gradient arriving from above is multiplied by σ' and passed down, layer after layer, and the product is what P4.3.2 computed. There is no algebraic partner to remove the factor.

Which is also why MSE with a sigmoid output is wrong. That pairing reintroduces the factor at the output — ∂L/∂z = 2(ŷ−y)ŷ(1−ŷ) — and Worked 4.1c measured the cost as a factor of 50 at ŷ = 0.01. So the rule is not "sigmoid is bad" but "sigmoid is fine wherever its derivative cancels, and nowhere else".

The link to 2.4.6. The sigmoid is on the output because it produces a number in (0,1) that can be read as a probability, which is what made calibration, ROC curves and cost-derived thresholds possible in Unit 2B. But saturation means a hidden layer's large pre-activation becomes an output of 0.9975, and section 2.4.6's warning applies with force: networks are systematically overconfident, and temperature scaling — dividing the logit by a fitted T — is the standard repair. It is Platt scaling, it is fitted on held-out data, and because it is monotone it leaves the AUC untouched.

Two things differ from Worked 4.4. The loss starts lower, 0.3133 against 0.6931, because these weights already lean the right way for B. And unit 2 sits exactly at z = 0, where ReLU is not differentiable — the convention ReLU'(0) = 0 is used here, so unit 2 receives no gradient from this example. Using ReLU'(0) = 1 instead would give δ₁ = (−0.2689, +0.2689). Both are defensible; state which you used.

Notice also that the zero column has moved: for point C it was column 2, for point B it is column 1, and in each case the zero sits where the corresponding input was 0.

Batch norm adds 2.8% to the parameter count, which is why its cost is never the reason to leave it out. Note also that a BN layer immediately after a dense layer makes that dense layer's bias redundant — BN subtracts the mean, which cancels any constant — so libraries typically set use_bias=False there, saving 150 parameters here.

nan means an arithmetic operation produced an undefined value, and in a training loop there are only a few ways that happens. The overwhelmingly likely cause here is exploding gradients: the backward recurrence multiplies eight weight matrices, and if their scale is above 1 the product grows geometrically, the update overflows to inf, and inf − inf on the next step gives nan. A learning rate of 0.5 with eight layers and no normalisation is asking for this.

Diagnose in this order, because each step is cheap and rules out a class of cause. First log the gradient norm every step; if it climbs across epochs 1 and 2, the diagnosis is confirmed before epoch 3. Second, check the loss function for log(0) — a cross-entropy implemented as −ln(ŷ) without clipping produces inf the moment a saturated sigmoid returns exactly 0 in floating point, and this is a genuinely common bug independent of the gradient issue. Third, check the data for nan or inf in the inputs, which propagates instantly and is embarrassing to discover late.

The fixes, in order of what to try. Drop the learning rate by a factor of ten — this alone resolves most cases and costs one run. Add gradient clipping, capping the global gradient norm at something like 1.0, which is the standard defence and is one line. Add batch normalisation, which keeps pre-activations in a controlled range. Check initialisation against 4.7's He scaling, since eight layers with weights that are slightly too large is exactly the setup for this. And use the fused "from logits" loss, which avoids the log(0) path entirely.

What was already there. Section 1.4 gave gradient descent and the chain rule. Section 2.3.1 derived ∂L/∂w = (ŷ − y)x for logistic regression, which is exactly this unit's output-layer equation with no hidden layers. So for a network of depth zero, backpropagation and 2.3.1 are the same three lines of algebra.

What backpropagation adds is one idea: reuse. The chain rule alone would let you compute ∂L/∂w for each weight separately, each time re-traversing the network from the loss down to that weight. Backpropagation notices that all weights in a layer share the same upstream factor δ⁽ˡ⁾, computes it once, and reuses it. That converts O(P) passes into O(1) and is the entire reason training deep networks is feasible. It is dynamic programming applied to the chain rule.

What it does not solve, and this is the more useful half of the answer. It gives you the exact gradient at the current point and nothing more. It does not make the loss surface convex — the XOR network succeeds from only 16.5% of random starts with two hidden units, so local minima and saddle points remain, exactly as in 3.1.1's k-means. It does not choose a step size, which is section 4.5's problem and which Worked 4.4 showed can kill a unit in one step. It does not prevent the gradient from vanishing or exploding as it passes through many layers, which is 4.3's problem and 4.7's. And it says nothing about generalisation: a perfect gradient on the training loss will drive you to a solution that may not transfer at all, which is 1.6's material and section 4.6's.

So the honest summary is that backpropagation solved the computational problem completely and left every optimisation and statistical problem untouched. The remaining five sections of this unit are about those.

The stability condition is |1 − 2η| < 1, so 0 < η < 1 here, and the fastest convergence is at η = 0.5 where the factor is exactly 0. In general for curvature L the condition is η < 2/L and the optimal rate is 1/L. The alternating sign is the signature of an overlarge learning rate, and a loss that oscillates upward between epochs is the same symptom in practice.

The practical property is that η sets a trust region rather than a sensitivity. With SGD, η multiplies the gradient, so you cannot choose it without knowing the gradient's typical magnitude — and that magnitude differs by orders of magnitude between an embedding layer and an output layer, and changes during training. With Adam, η bounds how far any single parameter can move in one step, roughly regardless of its gradient.

That is why Adam's default of 0.001 transfers across architectures while SGD's learning rate must be re-tuned for each one, and it is the main reason Adam is the default for anything new. The cost is the one Worked 4.5b measured: the discarded magnitude information was sometimes worth having.

Cause 1: an exploding gradient or a bad batch. A single outlier or a rare large gradient produced an update that threw the weights far from the good region. Distinguish by logging the per-step gradient norm — a single enormous spike at the epoch where the loss jumped confirms it. Fix with gradient clipping, which caps the norm and costs nothing.

Cause 2: the learning rate is marginally too high and it finally bit. Near a minimum the curvature rises, so a rate that was stable in a flat region becomes unstable in a sharper one and the run escapes. Distinguish by whether the loss oscillates before the jump rather than jumping cleanly, and by re-running with η halved. A decay schedule is the proper fix.

Cause 3: numerical, not optimisational. A log(0) in the loss, a division by a near-zero variance in batch norm, or overflow to inf. Distinguish by checking whether the loss is exactly inf or nan rather than merely large, and by checking whether the weights contain non-finite values. This is P4.4.3's territory.

And what to do regardless. Checkpoint every epoch, so a run that destroys itself at epoch 21 does not also destroy epochs 1 to 20 — restarting from the epoch-20 checkpoint with a lower rate usually just works. A run without checkpoints is the actual failure here.

What they change: the path, and therefore the speed. Section 1.4's gradient descent chooses a direction from local information only. Momentum adds a memory of previous directions, and Adam adds a memory of previous magnitudes. Both are ways of using the history of the optimisation to compensate for the fact that a single gradient says nothing about curvature. They are cheap approximations to what a second-order method would do properly, and they are used instead of second-order methods because computing a Hessian for a million parameters is not possible.

What they leave untouched: the landscape. None of them changes the loss surface, so none of them changes where the minima are. If the objective is non-convex, all three can land in a poor local minimum, and the XOR network's 16.5% success rate at two hidden units is the same for all three. This is exactly 3.1.1's k-means situation: Lloyd's algorithm converges reliably to a local optimum, and the fixes were more restarts and better initialisation, not a better descent rule. The same two fixes apply here and are sections 4.7's subject.

And a difference from k-means worth noting. In very high dimensions, the bad stationary points of a neural network loss are mostly saddle points rather than local minima, because a point is a local minimum only if the curvature is positive in every one of a million directions, which is vanishingly unlikely. Saddles are what momentum is particularly good at escaping: the gradient is near zero there, so plain SGD stalls, while accumulated velocity carries momentum through. That is a second, independent reason momentum helps, and it does not appear at all in the two-dimensional example of Worked 4.5b.

The surviving activations are doubled, which looks wrong until you notice that half the units are missing: the expected total is preserved, which is the only thing the next layer can rely on. Note also that the variance is not preserved — it is a²(1−p)/p per unit, which is 16 for the unit with a = 4 — and that injected variance is precisely the regularising effect.

That recovery property is why batch norm cannot hurt representational power: the identity transformation is inside its hypothesis space, so the network can always learn to switch it off. Compare this with dropout, which genuinely removes capacity and cannot be undone by learning — a real difference between the two, and the reason batch norm is often described as an optimisation aid that happens to regularise, rather than as a regulariser.

Dropout worked as designed and it did not help, which tells you the diagnosis was wrong. The training accuracy fell by six points, so the regularisation is definitely biting. Validation moved by one point, which is within noise on most validation sets. So the 28-point gap was not caused by the kind of overfitting dropout addresses.

Three more likely explanations. The first is distribution shift: if the validation set differs systematically from training — different hospital, different time period, different collection process — then no amount of regularisation closes the gap, because the model is being asked a different question. Check by evaluating on a held-out slice of the training distribution; if that scores 97%, the problem is shift, not variance.

The second is label noise or leakage. If training labels are cleaner than validation labels, or if a feature leaks the target in training only, this pattern appears exactly.

The third is simply not enough data for the task, in which case dropout at 0.5 is too blunt an instrument and the answer is data augmentation, transfer learning from a pretrained model, or a smaller architecture.

What to do next, in order. Plot a learning curve against training-set size, which section 4.8 covers — if validation accuracy is still rising at the full dataset size, more data is the answer and everything else is a distraction. Check for shift and leakage before tuning anything further. And do not stack more regularisers on top of a diagnosis that has already been falsified.

L2 weight decay is ridge regression, unchanged. Section 2.2's penalty λ‖w‖² adds 2λw to the gradient, so each step becomes w ← w(1 − ηλ') − ηg — a multiplicative shrink toward zero before the gradient step. Nothing about networks changes it. The name "weight decay" describes the same operation from the update's point of view rather than the objective's.

Dropout is bagging with shared weights. Section 2.3.5's random forest built many models on resampled data and averaged them, and the variance reduction came from decorrelating the members. Dropout trains 2ᵏ sub-networks that share parameters and approximates the average at test time by using the full network. The novelty is not the ensemble idea — it is doing it for free, inside one model, without training anything extra.

Early stopping is a constraint on distance travelled. For a linear model trained by gradient descent from zero, stopping after T steps is provably close to ridge with a particular λ that decreases as T grows — so it is an implicit L2 penalty applied by the optimiser rather than the objective. Same effect, no penalty term, and it is why "just train less" is a real answer to overfitting.

Batch normalisation is the genuinely new one, and it is not really a regulariser. Nothing in Units 1 to 3 resembles it, because it has no analogue in a model that processes one example at a time: it makes each prediction depend on the other examples in its batch. Its main effect is on optimisation — higher learning rates become safe, initialisation matters less — and its regularising effect is a side-effect of the batch-to-batch noise. Classifying it with dropout and L2 obscures what it does.

The unifying view from 1.7. Every one of these trades bias for variance, and the mechanism is always a restriction: on weight magnitude (L2), on co-adaptation (dropout), on distance from initialisation (early stopping), on activation scale (batch norm). The bias-variance decomposition is the same one that governed 2.2's ridge and 3.1.1's cluster shapes, and it does not become a different subject because the model has layers.

The two differ by only 12% here, because nᶜᵘᵗ is small relative to nᵢₙ. They diverge when the fan-in and fan-out are very unequal: for a layer with 10 inputs and 1000 outputs, He gives sd = 0.447 and Xavier gives 0.0445, a factor of ten. That is when the choice actually matters, and it is also when you should think about which direction — forward signal or backward gradient — you most need to preserve.

Let the two units have identical parameters at step t: w₁ = w₂ and b₁ = b₂. Show they are identical at step t+1.

The empirical confirmation is in section 4.7's opening: 4000 epochs from an all-0.5 start leaves W₁ = [[0.351, 0.351], [0.351, 0.351]] — the values moved, but the two rows stayed identical, so the layer never became more than one unit wide.

Note precisely what the proof needs: the incoming weights equal and the outgoing weights equal. Breaking either one is enough. This is also why dropout incidentally helps — it applies a different random mask to each unit, which breaks the symmetry even from a degenerate start.

2.30 is ln 10. That is precisely the cross-entropy of a model that outputs a uniform distribution over ten classes — it has learned nothing at all and is predicting 1/10 for everything. This is the ten-class analogue of ln 2 = 0.693 for binary, which appeared in Worked 4.4, and recognising these numbers instantly is worth doing: a loss sitting exactly at ln K means no learning, not slow learning.

With 30 layers and a stuck loss, the first suspect is the signal never reaching the output or the gradient never reaching the input. Check the initialisation scale against He: if Xavier was used with ReLU, the variance falls by 2³⁰ ≈ 10⁹ across the depth, and the output is numerically constant regardless of the input. Verify directly by feeding two very different inputs and comparing the logits — if they are identical to several decimal places, the forward signal has died.

Then check the other candidates. Log the gradient norm per layer; a norm that decays geometrically toward the input confirms vanishing gradients. Check for dead ReLUs, per P4.3.3. Check the learning rate is not so small that nothing moves, and not so large that it diverged immediately to a degenerate solution. And check the labels are actually being passed correctly, because a shuffled or constant label array produces exactly this symptom and is embarrassing to find on day three.

The fixes for a 30-layer network are structural. He initialisation, batch normalisation, and residual connections — the last of which is Unit 5's central mechanism and exists precisely because 30 plain layers do not train no matter how carefully you initialise them.

The protocol is identical to 2.1's and nothing about networks changes it. Split the data, tune on validation, touch the test set once. Every number used to select a setting — including the epoch chosen by early stopping — is a validation number and cannot be reported as a performance estimate.

What is harder than 2.1: cost and interaction. Fitting a logistic regression takes a second, so a 100-point grid search is free. Training a network takes hours, so the search budget is perhaps five or ten configurations, and that scarcity is what makes random search and successive halving necessary rather than merely clever. The hyperparameters also interact strongly — batch size with learning rate, depth with initialisation, regularisation strength with capacity — so tuning one at a time is not reliable in the way it roughly is for ridge's single λ.

What is easier than 3.1.1: there is an answer. Section 3.5's central difficulty was that unsupervised learning has no external referent, so "better" had to be assembled from geometry, stability and a null baseline. Here validation loss is a real measurement against real labels, and a lower one is genuinely better. That is a large advantage and it is worth appreciating after Unit 3.

What is shared with 3.1.1: non-convexity, and the same two fixes. Worked 4.7's 16.5% success rate is the same phenomenon as k-means converging to a poor local optimum from 28 of 220 starts, and the responses are the same — better initialisation (He here, k-means++ there) and more restarts. The distinctively deep-learning addition is a third fix unavailable to k-means: over-provision the capacity, because a wider network has an easier loss surface even though it represents no more than the narrow one needed to.

A gap of 0.33 that is still widening is variance, plainly. All three interventions below trade training loss for validation loss, which is the point.

A fourth option worth naming: a smaller model. It reduces variance but adds bias, so it is the least attractive of the four and is worth trying only after the others. And note that all four are 1.6's bias-variance trade-off with different labels — none of them is specific to neural networks.

Effect on accuracy: none, exactly. Dividing every logit by the same positive T preserves their order, so the argmax is class 1 in both cases. Accuracy, AUC, precision and recall at any rank-based threshold are all unchanged — which is the property that makes temperature scaling safe to apply after training.

Effect on ECE: potentially large. If this model is right about 74% of the time when it outputs 0.936, then T = 1 contributes a gap of about 0.20 and T = 2 contributes about 0.00. Worked 4.8's table had exactly this problem in its top bin.

The value of T is found by minimising negative log-likelihood on a held-out set — a one-dimensional optimisation over a single parameter, which takes seconds and is the cheapest meaningful improvement available to a trained network. T > 1 softens an overconfident model; T < 1 sharpens an underconfident one, which is rarer and is the random-forest failure noted in 2.4.6.

Almost everything needed to evaluate the claim, and the difference itself is probably not real.

No variance estimate. A single run of a single seed. Worked 4.7 showed seed-to-seed variation can be the difference between success and total failure; even on a well-behaved task, run-to-run standard deviation of a few tenths of a percent is normal. A 0.4-point gap with no spread reported could easily be one seed's luck. Report mean and standard deviation over at least five seeds, and compare with a paired test on shared splits, per 2.4.6.

No baseline and no prevalence. If the majority class is 94%, then 96.2% is a two-point gain, not a 96-point achievement. Section 2.4.2's rule applies verbatim.

No statement of how the test set was used. If the architecture, learning rate and epoch count were selected by looking at test accuracy, then 96.2% is a selection statistic and the honest estimate is lower. This is the most common way state-of-the-art claims become unreproducible, and it is 2.1's rule being quietly broken.

No accuracy-adjacent reporting. Per-class recall, a confusion matrix, calibration, and disaggregation by any group the model will be applied to. And no architecture, parameter count, training budget or seeds, so nobody can reproduce it — a model that reaches 96.2% using 100× the compute of the 95.8% baseline has not made the same kind of progress.

Unchanged: all of the arithmetic. The confusion matrix, precision and recall, ROC and AUC, cost-derived thresholds, regression metrics, ECE and Brier, and the whole of 2.6's fairness algebra including the impossibility proof. None of it ever depended on how the predictions were produced, so a network's scores are audited exactly as a logistic regression's were. The reporting discipline — validation for selection, test once, spread not point estimates — is also unchanged.

Worse: calibration and interpretability. Networks are more overconfident than the models of Unit 2, for identifiable reasons — large capacity trained to near-zero loss, batch normalisation, weak weight decay — so 2.4.6's warning applies more forcefully rather than less. And where a logistic regression offered one coefficient per feature, a network offers no faithful per-feature summary at all, so 2.6's interpretability requirement becomes something you approximate rather than something you read off.

Worse in a second way: variance across runs. Unit 2's models were mostly deterministic given the data. A network's result depends on the seed, and Worked 4.7's 16.5% is the extreme version. This makes single-run reporting actively misleading in a way it was not before.

Genuinely new: the epoch dimension, and double descent. Unit 2's models were fitted, not trained over time, so there was no "when to stop" question and no epoch curve to read. Early stopping, learning-rate schedules and the whole diagnostic vocabulary of section 4.8's first half have no Unit 2 analogue. And double descent means 1.6's U-shaped curve, which was reliable for every model in Unit 2, is no longer a safe guide at large capacity — the one place where a Unit 1 result genuinely needs qualifying rather than merely restating.

If the biases are removed from the two hidden layers, as is standard when batch norm follows, subtract 128 + 32 = 160, giving 12,773. Say which convention you used; both are marked correct if stated.

The trap is computing an update reflexively. The perceptron rule is error-driven: a correct prediction produces e = 0 and the weights do not move, however close to the boundary the point sits. Contrast this with gradient descent on cross-entropy, which would produce a non-zero gradient here because σ(0.5) = 0.622 ≠ 1 — the perceptron is satisfied by being right, and logistic regression is not.

The general result: on a constant patch of value v, a kernel outputs v × (sum of weights) + b. So a zero-sum kernel is blind to absolute brightness and responds only to variation, while a unit-sum kernel preserves brightness. This is why edge detectors are built to sum to zero and why a network trained on images with varying exposure tends to discover zero-sum filters in its first layer.

ReLU is worst here, and by a category rather than a margin. A gradient factor of exactly 0 means this unit passes nothing back and, if z < 0 for every training example, will never recover. The other three all pass something: leaky ReLU 0.01, tanh 0.0098, sigmoid 0.0452. Small is recoverable; zero is not.

Notice the reversal against the usual ranking. On the positive side ReLU is the best of the four, with a factor of exactly 1 where the others saturate. The activations differ not in quality but in where they fail, which is why the answer to "which activation is best" is always "for which regime".

The gradient pushes the true class's logit up (its component is negative, so the update z ← z − η∂L/∂z increases it) and both others down, with the largest push against class 2, which is currently the most confident wrong answer. The components summing to zero is a consequence of softmax's shift-invariance and is a fast sanity check.

Both columns are non-zero here because both inputs are non-zero, unlike Worked 4.4 where x₂ = 0 zeroed a column. And the second column is exactly twice the first, because x₂ = 2x₁ — the outer-product structure means the columns of ∂L/∂W are always proportional to the input.

The Adam step is exactly η, independent of the curvature and of the gradient being 6 rather than 2 — the property proved in P4.5.2. Note the practical consequence: SGD's safe rate depends on L, which you do not know and which changes during training, while Adam's does not. That is the trade the whole optimiser section is about.

Textbook high variance: 700 parameters per training example. The rising validation curve alongside a falling training curve is the signature, and the gap of 0.47 is large. Section 1.6's decomposition applies directly.

Fix 1: early stopping, restoring the best epoch's weights. The validation minimum was twelve epochs ago and everything since has been damage. This costs nothing, requires no retraining decision, and should have been on from the start.

Fix 2: reduce the effective capacity. With 700 parameters per example, this model can memorise the training set outright. Add dropout and L2, and consider a genuinely smaller architecture — if it is a CNN, check whether a flatten-to-dense layer holds most of those 2.1 million parameters, per Worked 4.2b, because global average pooling would remove them at a stroke.

Fix 3: more data, or the closest available substitute. Data augmentation if the domain permits it, and transfer learning from a pretrained model, which is usually the single most effective intervention at this data scale because it replaces most of the parameters with ones already fitted elsewhere.

Before any of that, plot loss against training-set size. If validation loss is still falling at n = 3{,}000, more data will help and is worth the cost; if it flattened at 1,500, it will not and fixes 1 and 2 are the whole answer.

A factor of 4096 will not overflow anything, so the network will train — badly. The symptom is very slow early progress and a strong dependence on the learning rate, rather than an outright failure, which makes it harder to diagnose than a total collapse. At 30 layers the same error gives 2⁻³⁰ ≈ 10⁻⁹ and the network does not train at all, which is P4.7.3.

Note that this is exactly Xavier's scaling used with ReLU, so it is a mistake made by using a sensible-sounding default rather than an arbitrary number.

Predictive parity nearly holds and equal opportunity fails badly. A flag means almost the same thing for both groups — 0.84 against 0.83 — so anyone checking "is a positive prediction equally reliable?" would find the model fair. But a qualified member of group B is found 52% of the time against 91% for group A, so nearly half of group B's qualified candidates are being missed. The 89% pooled accuracy reports neither number.

This is 2.6's impossibility theorem in its most uncomfortable form, and section 4.8 established that nothing about a network changes the algebra. If the base rates differ between the groups, equal PPV and equal TPR cannot both hold, so the near-equality of PPV is not evidence of fairness — it is the thing that forced the TPR gap.

What to do. First, decide which criterion the situation demands, and justify it: for a benefit-allocating decision like hiring, equal opportunity is the one that matters, so this model is indefensible as it stands. Then look for the mechanism — group-correlated proxies in the features, per 2.6, remembering that a network reconstructs a removed attribute more easily than a linear model. Consider group-specific thresholds, which close the TPR gap directly and are legally fraught in some jurisdictions, so this is a decision for people beyond the modelling team. Check whether the training labels themselves encode past discrimination, because a model that faithfully reproduces biased historical hiring is working exactly as trained.

And do not reach for interpretability tools as a defence. Section 4.8's warning applies: saliency and attribution methods on a network are approximations whose reliability is contested, and "we inspected the model and it looks fine" is not a finding. The disaggregated metrics are the evidence, and they are already damning.