The mathematics every model runs on

The dataset this whole chapter uses

One dataset, eight sections. Every formula gets applied to these five rows, so that by section 1.8 you already know the numbers by heart and can spend attention on the new idea instead of on new data.

Five sums are worth memorising now, because six sections use them: n = 5, Σx = 15, Σy = 25, Σx² = 55, Σxy = 88. The pass label is defined as marks 5 or above, which is why C, D and E pass.

The question

You have five students and you want one straight line through their marks. A straight line has two unknowns: where it starts and how steeply it climbs. Five data points give you five demands on those two unknowns. Five demands, two dials — you cannot satisfy them all.

So the honest question is not "what is the solution" but "what does an unsolvable system tell us, and what should we return instead". Every regression, every least-squares problem, and the closed-form solution used by linear regression in Unit 2 is this question.

The intuition

A single linear equation in two unknowns, like w₀ + 3w₁ = 5, is a line in the plane of possible parameter values. Every point on that line is a parameter pair satisfying that one equation. Two such equations are two lines, and solving the system means finding where they cross.

That picture immediately gives the three cases. Two lines that cross at a point: exactly one solution. Two parallel lines: no solution, the demands contradict each other. Two lines lying on top of each other: infinitely many solutions, because the second equation added no new information.

Machine learning lives almost entirely in the "no solution" case, and it is not a failure. It means the data is richer than the model. The fix is to stop demanding equality and start minimising disagreement.

The formal treatment

Write the system as A θ = b, where A is the m × d coefficient matrix, θ (theta) is the d-vector of unknowns, and b is the m-vector of right-hand sides. Here m counts equations and d counts unknowns.

The rank of a matrix is the number of genuinely independent rows in it — rows that are not sums or multiples of the others. [A | b] is the augmented matrix, formed by gluing b on as an extra column. If gluing b on raises the rank, then b points somewhere the columns of A cannot reach, and the system is inconsistent.

With m = 5 equations and d = 2 unknowns the system is overdetermined, and for real measured data it is inconsistent with probability one. Rather than solve it, we minimise the squared length of the residual vector Aθ − b. Setting the derivative of that squared length to zero gives the normal equations:

Three facts about this. The normal equations are always consistent, so a least-squares answer always exists. The matrix AᵀA is d × d — small, no matter how many data rows you have. And AᵀA is invertible exactly when the columns of A are linearly independent, which in data terms means no feature is a linear combination of the others.

Cost

Forming AᵀA costs O(md²) and inverting it costs O(d³). Linear in the number of data rows, cubic in the number of features. That is why the closed form is used for a 12-feature house-price model and gradient descent for a model with a million parameters.

The worked example

The visualization

The pitfalls

Practice

The question

Section 1.4 will minimise a function by repeatedly stepping downhill. When does that guarantee the best answer, and when does it merely find a comfortable spot? The answer depends on a single geometric property of the function, and the whole vocabulary of "the loss surface" rests on it.

The intuition

Picture two landscapes. The first is a smooth bowl. Drop a marble anywhere; it rolls to the one lowest point. Where it started does not matter, and once it stops you know it is at the bottom.

The second is an egg carton. Drop a marble and it settles in whichever dip it happened to land near. It has stopped moving, but there is a deeper dip three positions over that it will never find. Where it started decided everything.

Bowls are convex. Egg cartons are non-convex. Linear and logistic regression give you bowls; neural networks give you egg cartons in a million dimensions. That single distinction explains why linear regression is reproducible and neural network training is not.

The formal treatment

A set is convex if the straight segment joining any two of its points stays inside the set. A function f defined on a convex set is convex if its graph never rises above any of its own chords — where a chord is the straight line joining two points on the graph.

Two equivalent tests are easier to apply. The first-order condition: a differentiable f is convex exactly when its graph lies above every tangent line, so a tangent is a global underestimate. The second-order condition: f is convex exactly when its second derivative is non-negative everywhere, or in several variables when its Hessian — the matrix of all second partial derivatives — is positive semi-definite.

The payoff is one theorem, and it is the reason convexity is taught at all:

So on a convex loss, "the gradient is zero" means "done, and this is the best possible". On a non-convex loss it means only "locally flat", which a saddle point also satisfies.

The worked example

A non-convex function to compare against

Take g(w) = w⁴ − 4w² + w. Its derivative g′(w) = 4w³ − 8w + 1 has three roots, and g″(w) = 12w² − 8 is negative between ±√(2/3) ≈ ±0.816 — so the second-order test fails there and the function cannot be convex.

Both minima satisfy g′(w) = 0. Both are places where descent stops and reports success. Only one is right, and which one you get is decided by initialisation — which is exactly why neural network training in Unit 4 cares so much about how weights are initialised.

The visualization

The pitfalls

Practice

The question

Section 1.1 minimised squared error without ever justifying the square. Why not the absolute error? Why not the largest single error? Each choice is a different definition of a good model, and each one produces a different fitted line from the same five students.

The loss function is where you tell the algorithm what you care about. It is the single most consequential modelling decision in this unit, and it is the one usually made by copying whatever the tutorial used.

The intuition

Think of a loss as a fine charged per mistake. A flat fine of ₹100 per wrong prediction, regardless of size, treats a near miss and a wild miss alike. A fine proportional to the size of the miss treats a 4-mark error as four times as bad as a 1-mark error. A fine proportional to the square of the miss treats it as sixteen times as bad.

That last choice has a consequence worth seeing before any formula. Squaring makes the model terrified of large errors, so it will accept several small errors to avoid one big one. If a big error is a genuine emergency — a bridge load estimate — that is exactly right. If it is a typo in the data, the model will contort itself to accommodate a value that was never real.

The formal treatment

Three words that exams use interchangeably and this chapter does not:

Regression losses

MSE is differentiable everywhere and strictly convex, which is why closed forms exist for it. MAE is convex but has a kink at r = 0 where the derivative jumps from −1 to +1, so gradient methods need a subgradient and the optimum can be non-unique. Huber loss is the compromise: squared near zero so it is smooth, linear far out so a single outlier cannot dominate. The threshold δ is a hyperparameter.

Classification losses

The 0–1 loss is what you are actually graded on and it is useless for training: its gradient is zero wherever it is defined and undefined where it is not, so there is no downhill direction. Every practical classification loss is a smooth, convex surrogate for it. Log loss is the standard surrogate because it is convex in the model's linear score and because it punishes confident mistakes without limit — predicting 0.001 for a true positive costs 6.91, while predicting 0.4 costs 0.92.

Two ways to minimise

Set the gradient to zero and solve algebraically — possible only for a few losses and model families, as in 1.1. Or start somewhere and iterate downhill, which needs only that the loss be differentiable. That second route is section 1.4 and is how essentially every model past linear regression is trained.

The worked example

A log-loss calculation, since Unit 2 needs it

Treat the same students as a classification problem: predict pass. Suppose a model outputs the score z = −3 + 1.2x and converts it to a probability with the sigmoid p = 1/(1 + e⁻ᶻ) from section 1.8.

Only the probability assigned to the true class enters each row: for A the loss is −ln(1 − 0.1419) = 0.1530, for C it is −ln(0.6457) = 0.4375. All five students are classified correctly at a 0.5 threshold, so the 0–1 loss is 0 and accuracy is 100% — yet log loss is 0.2459, not 0. Log loss keeps grading confidence after accuracy has stopped noticing, which is why it is the training loss and accuracy is the reporting metric.

The visualization

The pitfalls

Practice

The question

Section 1.1 solved for the best line exactly, using a matrix inverse. That route closes the moment the model stops being linear in its parameters — no closed form exists for logistic regression, for an SVM, or for any neural network. What replaces it?

The intuition

You are on a hillside in thick fog and want to reach the valley floor. You cannot see the valley. But you can feel the slope under your feet, so you take a step in the steepest downhill direction, then feel again, and repeat.

Two decisions control everything. How do you measure the slope — carefully, by testing the ground all around you (expensive, accurate), or roughly, by one quick prod with a stick (cheap, noisy)? That is batch versus stochastic. And how big a step do you take? Too small and night falls before you arrive; too big and you stride straight over the valley and up the far slope. That is the learning rate, and it is the hyperparameter most likely to be the reason a model "does not work".

The formal treatment

For a function of several variables, the partial derivative ∂J/∂θⱼ is the rate of change of J when you nudge only θⱼ and hold the rest fixed. The gradient ∇J collects them into a vector.

Two properties justify the minus sign. The gradient points in the direction of steepest increase, and its magnitude is the rate of increase in that direction. So −∇J is steepest decrease, and for a small enough η the step is guaranteed to reduce J unless the gradient is already zero.

The gradient of MSE, derived

This derivation is examinable and worth being able to reproduce from nothing.

Setting that to zero recovers AᵀAθ = Aᵀy — the normal equations of 1.1. Closed form and gradient descent are not two methods for two problems; they are two routes to the same stationary point.

Batch, stochastic, mini-batch

One epoch is one full pass over the training data. Batch GD performs one update per epoch; SGD performs n. Comparing "100 iterations of batch" with "100 iterations of SGD" compares 100 passes with one fifth of a pass on this data — a mistake that makes SGD look far worse than it is.

The worked example

The visualization

The widget opens at η = 0.020 from the origin, so the first press of Step must reproduce Worked 1.4a exactly: (0.200, 0.704) with J = 8.156. Push η to 0.060 to see the zig-zag, and to 0.090 — past the 2/λₘₖₓ limit of 0.0845 — to watch it diverge.

The pitfalls

Practice

The question

Drop the marks and keep only the outcome: A and B failed, C, D and E passed. You are allowed one yes/no question about a student before guessing their result. Which question should you ask?

"Did they practise more than 2 hours?" tells you the answer outright on this data. "Did they attend the lab?" leaves you almost as unsure as before. To automate that comparison you need to measure "how unsure am I" as a number, and that number is entropy. Decision trees in Unit 2 are nothing but this comparison, repeated.

The intuition

Consider three bags of coloured balls. The first is all red: draw one and you knew in advance what you would get, so there is no uncertainty and learning the colour tells you nothing new. The second is half red and half blue: maximum uncertainty, and learning the colour is genuinely informative. The third is 90% red: you would bet red, and you would usually be right, so there is uncertainty but not much.

Entropy puts a number on that scale — 0 bits for the pure bag, 1 bit for the fifty-fifty bag, 0.469 bits for the 90-10 bag. One bit is exactly the information in one fair coin flip, which is what makes the unit intuitive.

The second idea is just as simple. Asking a question splits your group into two smaller groups. Compute the uncertainty of each smaller group, average them by size, and see how much the uncertainty dropped. That drop is the information gain, and the best question is the one with the largest drop.

The formal treatment

Four properties are worth knowing by name. Entropy is never negative. It is 0 exactly when one class has all the probability — a pure node. It is maximal, equal to log₂k, when all k classes are equally likely. And it does not depend on which class is which, only on the proportions, so a 30/70 split and a 70/30 split have identical entropy.

The weights |Sᵥ|/|S| matter and are the most commonly dropped part of the formula. A branch holding one sample out of a hundred is pure by luck; weighting stops that purity from counting for much.

Gini impurity, the alternative

Gini is the probability that two samples drawn at random from the node have different labels. It needs no logarithm, so it is cheaper, and in practice it selects nearly the same splits as entropy. CART uses Gini, ID3 and C4.5 use information gain. Where they disagree, entropy is slightly more willing to create small pure branches.

The worked example

The visualization

The pitfalls

Practice

The question

A degree-4 polynomial passes exactly through all five students. Its training MSE is 0.0000 — a perfect score by the measure section 1.3 built. Ask it about a student who practised 6 hours and it predicts −3.0 marks.

So a lower training error made the model worse. Any account of learning has to explain that, and the bias–variance decomposition is that account.

The intuition

Think about throwing darts, where the bullseye is the truth and each throw is a model fitted to a different sample of data.

All five darts tightly grouped, but in the top-left corner: consistent and consistently wrong. That is bias — the model is too simple to represent the truth, so no amount of data will centre it. Darts scattered widely all around the bullseye, averaging out to the centre: right on average, unreliable on any single throw. That is variance — the model is flexible enough to chase the particular noise in whichever sample it saw.

The trade-off is that the same knob controls both. Add flexibility and the grouping loosens while the centre improves. Remove it and the grouping tightens while the centre drifts. And under everything sits noise you cannot remove at all, because two students with identical hours can still score differently.

The formal treatment

Fix an input x. Imagine drawing a fresh training set, fitting the model, and predicting at x — then repeating that thousands of times. The prediction ŷ(x) becomes a random variable, and its expected squared error splits into three pieces that cannot interact.

Three consequences. Total error cannot fall below σ², so a reported test MSE of 0 on real data means a leak, not a triumph. Bias and variance are both non-negative, so neither can be traded for a negative amount of the other. And more training data shrinks variance without touching bias — which is why "get more data" fixes an overfitting model and does nothing at all for an underfitting one.

The worked example

The visualization

The pitfalls

Practice

The question

The degree-4 fit failed with coefficients of 17.5833 and −9.7917 on data whose values never leave the range 2 to 7. Section 1.6's remedy was to use a simpler model, which throws away capacity you might need elsewhere.

There is a better move. Keep the flexible model, and add a term to the objective that charges for large parameters. The model then uses its flexibility only where the data pays for it.

The intuition

Give the model a budget. It still wants to reduce error, but every unit of parameter magnitude now costs something, so it will only buy a large coefficient if the error reduction is worth the price. Wild cancelling coefficients — a huge positive next to a huge negative — are exactly what a budget kills first, because they cost a great deal and buy accuracy only inside the training range.

The strength of the budget is a single number, λ. At λ = 0 there is no budget and you are back to plain least squares. As λ grows, parameters shrink toward zero, and in the limit the model predicts a constant. Somewhere in between is the setting that generalises best, and finding it is a validation exercise, not an algebra one.

The formal treatment

Excluding the intercept is not a detail. Penalising w₀ would force predictions toward zero rather than toward the mean, so adding a constant to every mark would change the fitted model. Regularization is meant to constrain the model's shape, not its overall level.

Ridge has a closed form

Adding λ to the diagonal does two things at once. It shrinks every coefficient toward zero, and it makes the matrix invertible even when AᵀA is singular — because the smallest eigenvalue rises from 0 to λ. So ridge is simultaneously the cure for overfitting from 1.6 and the cure for the multicollinearity that broke 1.1. Historically it was invented for the second reason.

L1 versus L2, and why only one produces zeros

For a single centred feature, minimising SSE + λ|w| gives a soft threshold:

Ridge gives 13/(10+λ), which approaches zero but never arrives — at λ = 90 it is still 0.1300. Lasso subtracts a constant and therefore reaches zero at a finite λ, and stays there. That is why L1 performs feature selection and L2 does not.

Geometrically: the constraint region for L1 is a diamond with corners on the axes, and the first point of contact between an elliptical error contour and a diamond is usually a corner — a corner has one coordinate equal to zero. The L2 region is a circle, which has no corners.

The worked example

The visualization

Other ways to spend the same budget

Regularization does not have to be a term in the objective. Anything that restricts effective capacity does the same job, and Unit 4 uses all of these.

The pitfalls

Practice

The question

Everything so far output a raw number. Two things force a change. A probability must lie in [0, 1], and a raw score does not. And stacking layers must buy you something — which, as the next paragraph shows, it does not unless something non-linear sits between them.

The intuition

Compose two linear maps and you get a linear map. In symbols, with no activation between the layers:

A fifty-layer network with no activations has exactly the representational power of a single linear layer. All the depth in deep learning is bought by the non-linearity between layers, which is why this section exists and why it sits in the mathematical foundation rather than in Unit 4.

The activation is a squash applied element-wise to each unit's weighted sum. The choice matters for one reason above all others: backpropagation multiplies the activation's derivative once per layer, so a derivative that is habitually small makes deep networks untrainable.

The formal treatment

Note that both sigmoid and tanh have derivatives expressible in terms of their own output, which is why implementations cache the forward value and reuse it in the backward pass.

The numbers that decide everything

Read the σ′ column. Its largest value anywhere is 0.25, at z = 0. By z = ±4 it is 0.0177 — the unit is saturated, meaning its output barely responds to its input and its gradient has almost vanished. Backpropagation through k sigmoid layers multiplies k such numbers together:

That is the vanishing gradient problem, and it is why deep sigmoid networks were nearly untrainable before ReLU. ReLU's derivative is exactly 1 for all positive inputs, so it multiplies through any depth without shrinking anything.

Which, when and why

The worked example

A softmax calculation

Three-class scores z = [2, 1, −1]:

Shift invariance is why libraries subtract max(z) before exponentiating: it cannot change the answer and it prevents eᶻ overflowing for large scores.

The visualization

The pitfalls

Practice

The one table to memorise

Every number below is derived somewhere in this chapter from the same five students. If you can reproduce this column of results from n = 5, Σx = 15, Σy = 25, Σx² = 55, Σxy = 88, you have the unit.

Further reading

• Géron, Hands-On Machine Learning, 3rd ed., ch. 4 — the prescribed textbook. Best for seeing gradient descent, batch versus stochastic, and ridge/lasso as running code with plots. Light on derivations.
• Bishop, Pattern Recognition and Machine Learning, ch. 1.1, 1.5 and 3.1–3.2 — the careful treatment of the bias–variance decomposition and of regularization as a Gaussian prior. This is where to go when a derivation in this chapter felt compressed.
• Hastie, Tibshirani and Friedman, The Elements of Statistical Learning, ch. 3 — ridge and lasso in full, including the geometry of why L1 produces zeros.
• Goodfellow, Bengio and Courville, Deep Learning, ch. 4, 6.3 and 8.2 — conditioning and ill-conditioning, the activation function survey, and why non-convexity is manageable in practice.
• Mitchell, Machine Learning, ch. 3 — the original clear account of entropy and information gain in ID3, including gain ratio.