Six algorithms, one dataset, four boundaries

Fits: 3 models × 8 settings × 5 folds = 120. Each fit trains on 4 of the 5 folds, so 500 × 4/5 = 400 rows, and validates on 100.

Then one final fit on all 500 rows using the winning configuration, before the single test-set evaluation — 121 fits in total. That last refit is standard and often forgotten: cross-validation is used to choose, and the shipped model is retrained on everything the choice was allowed to see.

At k = 3 the unstratified estimate collapses to chance while the stratified one holds at 0.8750, on identical data with an identical model. The cause is the class balance of each training set. Holding out {C,D} leaves six rows that are 2 failures against 4 passes, so any three-row neighbourhood is more likely to be majority-pass simply by prevalence — and the rows being validated are the two failures. Holding out {E,F} does the mirror image.

Two lessons. Unstratified folds do not merely add noise, they add bias, and the bias points against whichever class the fold over-represents. And k = 3 is more exposed than k = 1, because a vote among more neighbours draws harder on the training prevalence. This is why stratification is the default rather than a refinement, and it is also a preview of the class-imbalance problem that Unit 2B is largely about.

The drop is expected, not reassuring, and the reasoning has the logic backwards. Taking the maximum over 40 CV scores is an optimistically biased estimate of the winner's true accuracy, because part of what made it the maximum was luck in the fold assignment. The test score of 0.88 is the honest number; the 0.91 was inflated by the selection.

What the pair does tell you is that the inflation was small, roughly 3 points, which suggests the folds were large enough that the selection did not overfit badly. What it does not tell you is that 0.88 is good, or that this model beats the runner-up — the runner-up might well test higher. And the one thing your colleague must not now do is try a 41st configuration and report its test score.

Cross-validation estimates expected test error, which by 1.6 is bias² + variance + σ². It sees only the sum. It cannot separate the three terms, because doing so requires many independent training sets drawn from the true distribution and knowledge of the true function — neither of which you have. The decomposition tells you what to do about a large CV error; it is the train-versus-validation gap, not CV itself, that hints at which term dominates.

Leave-one-out's variance has two sources. Its n training sets overlap in n−2 rows, so the n scores are strongly positively correlated and averaging them reduces variance far less than averaging independent scores would — the same ρ + (1−ρ)/B effect that limits ensembles in 2.3.5. And each individual score is a single 0-or-1 outcome, the noisiest possible measurement. Five-fold trades a little extra bias, from training on 80% rather than (n−1)/n of the data, for markedly less variance, which is why it is the usual default.

Model: ŷ = 3.5714 + 1.2857x₁ + 0.2857x₂. Check by the point of means: 3.5714 + 1.2857(3) + 0.2857(2) = 8.0000 = ȳ. Note the feature correlation here is 4/√100 = 0.4, appreciably stronger than the spine's 0.1581, so these two coefficients are more entangled than the spine's are.

The design matrix now has columns [1, x₁, x₂, x₁+x₂], and the fourth is the sum of the second and third. Its rank is 3 while d + 1 = 4, so AᵀA is 4×4 with determinant exactly 0 and no inverse. Ordinary least squares has no unique answer.

The predictions are unchanged: any (w₁, w₂, w₃) with w₁ + w₃ = 1 and w₂ + w₃ = 2 reproduces the identical plane, so (1, 2, 0), (0, 1, 1) and (−5, −4, 6) all fit exactly as well. The fit survives; the interpretation does not.

Ridge repairs this. Adding λ to the diagonal makes the matrix invertible for any λ > 0, and among the infinitely many equally good solutions it selects the one of smallest ‖w‖² — which spreads the effect across the correlated columns rather than concentrating it arbitrarily. That is Tikhonov regularization doing the job it was invented for, and it is why ridge is the default first response to multicollinearity.

The fit is probably fine and the coefficient probably means what it says: among houses of the same floor area, one with more bedrooms sells for less. That is plausible — it describes the same square footage chopped into more, smaller rooms, and buyers pay for space.

What has gone wrong is only the reading. Bedrooms and floor area are strongly correlated, so the marginal relationship between bedrooms and price is positive while the partial relationship, holding area fixed, is negative. Both are true statements about different questions. Regress price on bedrooms alone and the sign will flip.

Three things to do before quoting either number. Check the VIF for bedrooms; if it is 10 or more, the coefficient is estimated from very little independent variation and its value is unstable even if its sign is right. Consider a more interpretable parameterisation, such as floor area plus average room size. And never present a partial coefficient to a non-technical audience without stating what was held fixed, because "an extra bedroom costs you twelve thousand" is how a correct model becomes a false headline.

Predict a pass when 3 + x₁ + 2x₂ ≥ 12, that is x₁ + 2x₂ ≥ 9. Evaluating: A 1+6=7 fail, B 2+2=4 fail, C 2+6=8 fail, D 3+4=7 fail, E 3+8=11 pass, F 4+6=10 pass, G 3+10=13 pass, H 6+6=12 pass. All eight correct.

So on this data the regression route works, and it even gives a boundary — x₁ + 2x₂ = 9 — that differs from the x₁ + x₂ = 6 that 2.3.1 and 2.3.3 will find. Different because it was fitted to a different target: the marks, not the labels.

Three reasons not to make a habit of it. Least squares punishes a prediction of 19 for a passing student as heavily as a prediction of 5, although only one of those is an error — squared loss on a thresholded target charges for being too right. The output is not a probability and cannot be calibrated into one; nothing keeps it inside [0,1]. And a single extreme mark drags the boundary, because 1.3 showed how far one outlier moves a squared-error fit, whereas the classifiers of 2.3 would barely notice. Predicting the number is the better choice when you need the number; when you need the class, use a loss designed for classes.

So one more assignment multiplies this student's odds of passing by 3.32, taking the odds from 2.23 to 7.39, which is p = 0.8808. The probability rose by 0.19 while the odds more than tripled — a reminder that odds ratios sound larger than the probability change they describe.

The boundary is identical: 2s − 12 = 0 is still s = 6. Scaling the whole weight vector cannot move the boundary, because the boundary is where z = 0 and zero times anything is zero. What changed is confidence, and log loss fell by a factor of five to reward it.

This is why log loss can be driven arbitrarily low on separable data, and why accuracy cannot distinguish the two models at all. It is also the mechanism by which an unregularized fit becomes overconfident: the training loss genuinely improves, so gradient descent genuinely pursues it, and nothing in the objective knows that 0.9975 is an unjustifiable claim about a student it has seen once.

The odds ratio. Three and a half times the odds, not the probability, and only approximately equal when the outcome is rare. If baseline risk is 1%, odds are 0.0101, tripled-and-a-half gives 0.0354, so the risk becomes 3.4% — close enough that the manager's sentence is nearly right. At a baseline of 40%, odds 0.667 become 2.333 and risk becomes 70%, which is 1.75× the probability, not 3.5×. The rarer the outcome, the better the approximation. Also, the ratio is conditional on the model's other covariates being held fixed, and it says nothing about what would happen if the factor were removed.

The accuracy. For a rare complication, 96% accuracy is compatible with a model that never predicts a complication at all — if the base rate is 4%, predicting "no complication" always scores exactly 96%. The number is uninformative without recall, precision and the confusion matrix, which is precisely the material of Unit 2B. Ask for those before asking anything else.

Setting the gradient to zero gives, for linear regression, Aᵀ(Aθ − y) = 0 — linear in θ, so it rearranges into the normal equations and is solved by linear algebra. For logistic regression it gives Aᵀ(σ(Aθ) − y) = 0, where θ appears inside a sigmoid. That is transcendental; no rearrangement isolates θ, so the root must be found numerically.

Reproducibility survives because of 1.2. The log-loss objective is convex in θ, so every local minimum is global and any correctly implemented descent from any starting point reaches the same answer. Two runs that disagree indicate a bug, a learning rate above the stability limit, or insufficient iterations — not a legitimate second optimum. That guarantee is what disappears in Unit 4, where the surface is non-convex and two runs disagreeing is normal.

The one caveat is this section's depth box: on separable data the optimum is at infinity, so "reaches the same answer" holds for the direction of w but not its length, and where you stop is set by your iteration budget rather than by the mathematics. Regularizing removes the caveat.

k = 1: F and H tie at d² = 2 and both are passes, so the tie is harmless — pass.

k = 3: F (pass), H (pass), D (fail) → 2–1 → pass, vote fraction 0.667.

Sanity check against the linear models: R has x₁ + x₂ = 7, on the pass side of the boundary s = 6. All three methods agree.

k = 1: G, pass — unchanged. k = 3: G plus the tied pair E and C, so three voters — pass, pass, fail → pass 2–1, unchanged. k = 5: G, E, C, A and then a three-way tie at distance 4, so all of F, D and B join, giving seven voters: G, E, F pass against C, A, D, B fail → fail 3–4.

So the flip that Euclidean distance produced at k = 7 arrives at k = 5 under Manhattan. The reason is structural: on integer coordinates, Manhattan distance takes far fewer distinct values than squared Euclidean, so ties are much more common and any fixed k silently pulls in more neighbours than requested. The metric is a hyperparameter with real consequences, and on lattice-like data it also changes how often you must invoke a tie rule.

Reduce n. Prototype selection or clustering the training set down to representatives cuts the distance count directly and proportionally. Risk: rare classes and boundary cases are exactly what gets discarded, so accuracy loss is concentrated where accuracy matters.

Index instead of scanning. A KD-tree or ball tree gives sublinear search, but their advantage decays as dimension rises and at 30 features the gain over brute force is modest. Approximate nearest neighbour methods — locality-sensitive hashing, HNSW graphs — do work at this dimension and routinely give 10–100× speedups for a small, measurable recall loss. This is usually the right first attempt because it preserves the model.

Replace the model. Train a logistic regression, gradient-boosted trees or a small network on the same data. Prediction becomes microseconds and memory drops from gigabytes to kilobytes. You lose k-NN's ability to represent an arbitrarily complicated boundary with no assumptions, and you have to accept whatever accuracy the substitute reaches — but at 30 features a boosted-tree model very often matches or beats k-NN anyway, so measure before assuming a sacrifice.

The one option not to take is lowering k. Prediction cost is dominated by computing n distances, not by selecting k of them, so k = 1 is barely faster than k = 5 and considerably worse.

At k = 1 the nearest stored row to any training point is that point itself, at distance zero, so it votes for its own label and is always correct. Training accuracy is 1.0000 by construction for any dataset with no contradictory duplicates. In 1.6's vocabulary this is the degree-4 polynomial again: a model with enough capacity to interpolate reports zero training error as a structural fact, not as evidence.

Leave-one-out removes exactly that degeneracy. Holding out row i deletes it from the stored set, so the vote comes from a genuinely different student and the score measures generalisation. On the spine this gives 0.8750: seven of eight rows are classified correctly and student F is lost, because with F removed its two nearest remaining neighbours are D at d² = 2 and E at d² = 2 — one failure and one pass, a deadlock.

The bias–variance reading is that k = 1 sits at the extreme low-bias, high-variance end, and F is where that variance shows: F is one of the two passes hardest against the boundary, so its single nearest neighbour is as likely to come from the wrong class as the right one. Raising k trades that variance for bias, and 2.1 found that eight rows cannot resolve which trade is better — which is itself the honest finding.

By symmetry about the horizontal axis the boundary must be vertical, so w = (a, 0). The canonical conditions are 3a + b = +1 and a + b = −1. Subtracting gives 2a = 2, so a = 1 and b = −2.

The data is still separable — the failures now reach s = 6 and the passes start at s = 7 — but the corridor has narrowed from the interval (5, 7) to (6, 7). The boundary moves to the new middle, s = 6.5, and the normalisation must be redone:

One added point halved the margin and quadrupled the objective. The support vectors are now I, E and F; C and D have slack and drop out entirely. That is the fragility argument for soft margins made concrete: with C finite, the optimiser could pay a small hinge penalty to leave I inside the margin and keep most of the original width, which is almost certainly the better model. A hard margin has no such option.

96% of the data sits on or inside the margin, which means the model has found almost no clean separation. Two diagnoses, distinguished by the training accuracy. If training accuracy is high, C is large and γ is probably large too: the model has wrapped a tight bubble around each training point, and it will generalise badly — effectively a memorising k-NN with extra steps. If training accuracy is also poor, the kernel simply does not suit the data.

It is also a practical problem regardless of accuracy, because prediction cost scales with the number of support vectors: every query needs 4,800 kernel evaluations.

What to change, in order. Verify the features are scaled, since unscaled features are the most common cause. Reduce γ, which widens each kernel's reach and lets one support vector cover more ground. Reduce C, accepting more margin violations for a wider margin. Then re-tune the two jointly on a log grid, since they interact strongly. And compare against a linear SVM, which on many real datasets performs as well and gives a compact model.

The connection is the shape of the two losses. Hinge loss, max(0, 1 − ŷf), is exactly zero for any point with functional margin at least 1, so once the margin is achieved those points stop influencing the solution entirely. Log loss, ln(1 + e^(−ŷf)), is never exactly zero — but it decays exponentially, so a point at large margin contributes almost nothing. The two losses are close everywhere except near the boundary, and both therefore end up caring about the same handful of marginal points.

2.3.1's depth box supplies the exact statement: on separable data, unregularized logistic regression drives ‖w‖ → ∞ in the direction that maximises the minimum margin, which is by definition the hard-margin SVM solution. Agreement here is a theorem, not a coincidence.

They diverge when the data is not separable, and specifically when there are points far on the wrong side. Log loss grows linearly in the margin violation, so a badly misclassified outlier keeps pulling the boundary forever. Hinge loss also grows linearly, so a soft-margin SVM is pulled too — but its multiplier is capped at C, which places a hard ceiling on how much any single point can influence w. That cap is why SVMs are the more robust of the two to a handful of extreme mislabels. They also diverge in what they can deliver: logistic regression returns calibrated probabilities, and an SVM returns a distance that must be calibrated separately — so if the output feeds a threshold that will be tuned later, as in Unit 2B, logistic regression is the more convenient starting point.

The root split x₁ ≤ 2.5 gives branches of 3 and 5 rows, both at least 2, so it is still allowed and still the best. The left branch is pure and becomes a leaf.

On the right branch, the winning split x₂ ≤ 2.5 would produce branches of 1 and 4 rows. The one-row branch violates the constraint, so that split is forbidden. The alternatives, x₂ ≤ 3.5 and x₁ ≤ 3.5, both give branches of 3 and 2 rows with IG = 0.1710. Take x₂ ≤ 3.5: left is D, F, plus... on this branch the rows are D(3,2), E(3,4), F(4,3), G(3,5), H(6,3), so x₂ ≤ 3.5 gives left {D, F, H} = 1 fail 2 pass, predicting pass, and right {E, G} = pure pass.

So the constraint has bought nothing and cost a split. Two lessons. A minimum-leaf-size rule can force the tree into a split that is worse than not splitting at all, which is why cost-complexity pruning — grow first, then remove — is the better discipline. And on eight rows, every regularization setting collapses to roughly the same 0.8750 model, which is 2.1's finding restated: the dataset is too small to support a choice.

The gap. 0.998 against 0.71 is severe overfitting, exactly the pattern of 1.6's diagnosis table. Depth 12 permits up to 4096 leaves, so most leaves hold a handful of rows and encode noise.

customer_id as the top feature. Two problems at once. It is a high-cardinality identifier, so the depth box's bias guarantees it will look important whether or not it is. And it is almost certainly leakage: if the same customer appears in training and validation, the tree has memorised individuals rather than learning behaviour, and 2.1's warning about grouped data applies — the split should have been by customer, not by row. Drop the column.

timestamp as second. Same cardinality bias, plus a likely temporal leak. If the model will be used on future data, a raw timestamp lets the tree partition by "which batch this came from", which is unavailable and meaningless at prediction time. Replace it with derived features that will exist in future — hour of day, day of week, days since signup — and validate on a time-ordered split.

What to do. Remove the identifier and the raw timestamp, re-split by customer group and by time, cap the depth or tune cost-complexity α by cross-validation, set a minimum leaf size, and recompute importance by permutation rather than by impurity. Expect the training accuracy to fall a long way. That fall is the model becoming honest, not becoming worse.

The difference is the hypothesis space. The SVM's boundary is x₁ + x₂ = 6, and D sits at s = 5, a full 0.7071 units clear on the correct side — D is a support vector, so it defines the boundary rather than straining against it. The tree cannot express that diagonal at all. Its axis-aligned cuts place D in the same x₁ > 2.5 region as four passes, so the only way to get D right is a second cut isolating it alone.

In 1.6's terms, the tree pays for its restricted hypothesis space with bias, and then buys the last training point back with variance — a leaf built on one observation. The SVM pays no such price because the diagonal was in its space from the start. In 1.5's terms, information gain is a greedy one-feature-at-a-time criterion and is structurally blind to a rule that depends on a sum of features, which is why the root's best gain was only 0.5488 rather than 1.

Ship the SVM, or logistic regression. The generating structure here is a threshold on total effort, and both linear models represent it exactly with a wide margin, while the tree needs a memorised special case. If interpretability were mandatory, the honest tree is the depth-1 stump — "practise more than 2.5 hours" — at 0.8750, and you would say plainly that it misses one student. What you would not do is ship the depth-2 tree and cite its 100% training accuracy, which is the specific failure this whole unit exists to prevent.

Note that e^(−α) = √(ε/(1−ε)) in general, so the correct mass becomes √(ε(1−ε)) and so does the incorrect mass — identical by construction, which is precisely why the split is always 50/50. Also Zₜ = 2√(ε(1−ε)) = 0.8660 here, and since Zₜ < 1 whenever ε ≠ 0.5, the product of the Z values bounds the training error and drives it to zero exponentially. That bound is AdaBoost's convergence proof in one line.

For student D: learner 1 said pass (h₁ = +1, wrong) with weight 0.9730; learner 2 said fail (h₂ = −1, right) with weight 0.8959. The ensemble score is 0.9730(+1) + 0.8959(−1) = +0.0771, so H(D) = sign(+0.0771) = +1 = pass — still wrong, but only just.

That near-tie is instructive. One round of correction was not quite enough to overturn a confident first learner, and D's weight will now rise again for round 3, at which point a third learner agreeing with the second will flip the verdict. Boosting converges by accumulation, not by any single round winning, and it is why T in the tens or hundreds is normal even on easy problems.

Almost none. Use the variance formula. Going from 500 to 5000 changes (1−ρ)/B from (1−ρ)/500 to (1−ρ)/5000, and both are negligible beside ρ for any realistic tree correlation. At ρ = 0.3 the remaining variance moves from 0.30140 to 0.30014 — a change in the fourth decimal place, invisible against the sampling noise of the OOB estimate itself. The cost is ten times the training time and ten times the prediction latency.

What would actually help. Lower ρ by reducing m, the features considered per split, since that is the only term left that is not already exhausted — tune it on a small grid and watch OOB. Raise each tree's quality with better features. Or change model family: gradient boosting attacks bias, which a forest of already-averaged trees cannot reduce further, and on tabular data it frequently beats a random forest outright.

The general rule to take away: once the ensemble-size curve has flattened, more members is the one intervention guaranteed not to help.

A fully grown tree is close to unbiased and very high variance: it can represent almost any axis-aligned partition, so its average over training sets is near the truth, but any single tree depends heavily on which rows it saw. 1.6's decomposition says averaging leaves bias untouched and divides variance by up to B. A model with all its error in the variance term is therefore the ideal candidate, and a deep tree is the standard example of one.

On the spine it would fail, for two reasons. Bootstrap samples of eight rows contain on average only 5 distinct students, and 2.3.4 showed that the informative structure lives in a single row — student D. Any bootstrap sample missing D produces a tree with no reason to make the second split, and roughly a third of samples miss it. More fundamentally, the true boundary is diagonal, so every tree in the forest shares the same axis-aligned bias, and averaging cannot remove a bias that all members share. The forest would converge to a smoother staircase, not to the diagonal.

Boosting attacks bias. Each stump is a very high-bias model — one question, two leaves — and the sum of many weighted stumps is a far richer function than any of them. On the spine, an additive combination of stumps on x₁ and x₂ can approximate the diagonal to any accuracy, which is exactly the thing a single tree of bounded depth cannot do. The price, per this section's depth box, is that capacity now grows with T, so variance returns and T must be validated.

Identical numbers to Q, by an accident worth understanding: swapping "practice high, assignments low" for "practice low, assignments high" exchanges two factors that happen to be reciprocal in both classes, and swapping lab yes for lab no does the same. The lesson is that after discretisation Q and R are indistinguishable to this model, whereas in the original features they sit on opposite sides of the boundary x₁ + x₂ = 6 only if their totals differ. Discretisation destroyed the distinction.

The prior favoured ham four to one and the likelihoods overturned it decisively, because each word is roughly six to fifteen times more common in spam. Note how much work the independence assumption is doing: "invoice" and "urgent" almost certainly co-occur in real spam, so multiplying their evidence double-counts it and 0.9577 is too confident. The decision is still right, which is the depth box's point exactly.

Whether the prior is in the model at all. A prevalence of 0.0001 is an extremely strong prior against the disease. If the model was trained on a balanced dataset — a common and reasonable choice for learning the likelihoods — then its implicit prior is 0.5 and the reported posterior is wrong by four orders of magnitude in the prior term. Recomputing with the true prior can easily turn 0.99 into something under 0.1. The fix is to adjust the prior explicitly at prediction time rather than retraining.

Whether the symptoms are independent. In medicine they are usually strongly correlated — fever, fatigue and elevated white cell count travel together — so each is counted as fresh evidence when it is largely the same evidence. This pushes the posterior toward 1 and is the second reason 0.99 is not trustworthy.

What happens next. Even a correctly computed high posterior for a rare disease should trigger confirmatory testing, not treatment, and the honest output for a screening model is a risk score with a stated interval. The concrete request to make is for the confusion matrix at the deployed threshold, at the true prevalence, plus a calibration curve — which is exactly the toolkit Unit 2B builds.

What they model. Naive Bayes is generative: it estimates P(x | c) and P(c) and derives the posterior by Bayes' theorem. Logistic regression is discriminative: it estimates P(c | x) directly and never models the features at all. Gaussian Naive Bayes with a shared covariance across classes gives a linear boundary of exactly the form logistic regression fits, so the hypothesis spaces coincide while the fitting criteria do not.

How they are fitted. Naive Bayes needs one counting pass and has a closed form; logistic regression needs iterative optimisation, per 2.3.1. Naive Bayes accepts the independence assumption as a constraint and therefore converges to its best answer with very little data. Logistic regression makes no such assumption, so with enough data it reaches a strictly better fit — and with too little, its greater flexibility costs it in variance.

When to choose which. Naive Bayes when data per class is scarce, when features are numerous and sparse, when training must be instant or incremental, or when you want to reason explicitly about a changing prior — adjusting for a new prevalence is a one-line change. Logistic regression when you have enough data to afford the flexibility, when features are correlated, and above all when you need the output probability to mean something, since log loss trains calibration directly while Naive Bayes' independence violation destroys it.

(a) Logistic regression, or a shallow tree. A written reason requires a model whose decision decomposes into named factors; a signed coefficient with an odds ratio, or a three-question rule, both do. There is usually also a legal requirement behind this, which makes it a constraint rather than a preference.

(b) Multinomial Naive Bayes as the baseline, then linear SVM or logistic regression on TF-IDF features. Text gives thousands of sparse features where the independence assumption is wrong but harmless, and 50,000 rows across 12 classes is enough for a discriminative linear model to overtake it. Train the cheap one first so you know what beating it looks like.

(c) Regularized linear model, lasso or a linear SVM. With 400 features and 300 events, d > n: k-NN's distances are meaningless, a deep tree will memorise, and an unregularized fit has no unique solution. Lasso additionally tells you which sensors matter, which is likely the real deliverable.

(d) Gradient-boosted trees, thresholded on the cost-weighted metric. Prediction is a few hundred fast comparisons, comfortably inside 40 ms, and boosting handles the mixed tabular features fraud data usually has. The model choice is the easy part: at 0.2% prevalence, accuracy is useless, the threshold must be set from expected cost rather than left at 0.5, and class weighting is required — all of which is Unit 2B.

The midpoint has x₁ + x₂ = 6, exactly on the linear boundary. Logistic regression outputs p = 0.5000 — a formal refusal to decide, which is the correct answer for a point equidistant from both classes. The SVM gives wᵀx + b = 0, on the boundary, and its sign convention resolves to a class arbitrarily; the honest report is that it is undetermined.

The tree asks x₁ ≤ 2.5, which is true, so it answers fail immediately without ever consulting the assignment count. It is confident, and its confidence rests on a threshold placed midway between two students. k-NN at k = 3 finds C and E tied nearest at d² = 0.5, one failure and one pass, and then a four-way tie at d² = 2.5 among A, D, G and F. The third distance is 2.5, so every row tied at it votes: six voters, A, C and D failing against E, F and G passing, a 3–3 deadlock.

They disagree because fitting the training data perfectly constrains a model only at the training points. Everywhere else the prediction is determined by the model's inductive bias — the linear models interpolate along a straight line, the tree extends its axis-aligned cuts, k-NN follows the nearest rows — and the eight students say nothing about which bias is right. This is the deepest point in the file: the training data selects among models within a family, and the family itself is your assumption, not the data's conclusion. Choosing it is the modelling work, and validating that choice is 2.1.

Check through the point of means: 5.2 + 1.2(3) + 0.6(2) = 10.0 = ȳ. Note the feature correlation is 6/√96 = 0.6124, appreciably higher than the spine's, so these two coefficients are substantially entangled — a warning to attach to any interpretation.

An odds ratio below 1 means the feature reduces the odds — each extra unit of x₂ multiplies the odds by 0.6065, a 39% reduction. At threshold 0.6 this prediction is marginal: one more unit of x₂ takes z to 0 and p to exactly 0.5, flipping the decision.

The closest opposing pair is (2,2) and (4,4), differing by (2,2), so the normal direction is (1,1) and the boundary is perpendicular to it. Using s = x₁ + x₂: the negatives have s = 2, 4, 3 and the positives s = 8, 9, 9. The corridor runs from 4 to 8.

Same boundary equation as the spine, x₁ + x₂ = 6, but a margin twice as wide because the classes are further apart — and correspondingly a smaller ‖w‖. The margin is a property of the data, not of the equation you write down.

Split A, by roughly four times. It buys a completely pure branch holding half the rows, and even though its other branch is left at maximum impurity, that is cheaper than B's two mediocre branches. The tree takes A and continues working on the impure side — which is how trees grow.

At ε = 0.55, α = ½ln(0.45/0.55) = −0.1003, which is negative. The learner is worse than chance, so the ensemble uses its prediction inverted, and it still contributes usefully. In practice most implementations stop instead: a learner that cannot beat 0.5 on the current weighting usually signals that the weak learner family is exhausted, and continuing adds noise. Note also that Z = 2√(0.55 × 0.45) = 0.9950 < 1, so even this round tightens the training-error bound slightly.

Averaging 200 trees has removed only half the variance of a single tree, and it is within 0.5% of everything it will ever remove. The B term is exhausted; adding trees is pointless.

The promising change is to reduce m, the number of features considered at each split, because ρ = 0.5 is high and m is the direct lever on it. Each tree will get slightly worse and the ensemble should get better — tune m on a small grid and watch OOB accuracy. Failing that, switch to gradient boosting, which reduces bias rather than variance and so attacks the part a forest cannot.

k-NN — changes it drastically. Squared Euclidean distance adds (Δx₁)² to (Δx₂)², and the second term is up to 10⁶ times larger, so x₁ is effectively deleted. Standardising is mandatory.

Decision tree — no change at all. Splits depend only on the ordering of a feature's values, and standardising is monotone, so every candidate threshold maps to an equivalent one and the tree is identical. This is one of very few models where scaling genuinely does not matter.

Ridge regression — changes it. The unpenalised fit is unaffected in its predictions, since a linear model absorbs a rescaling into the coefficient. But the penalty λ(w₁² + w₂²) charges both coefficients equally, and x₂'s natural coefficient is roughly a thousandth of x₁'s, so it is barely penalised. The choice of units silently decides which feature is shrunk. Standardise.

RBF SVM — changes it, and worst of all. The kernel exp(−γ‖x − x′‖²) uses the same distance k-NN does, so x₁ is ignored, and a single γ now has to suit two wildly different scales, which it cannot. Standardise before tuning γ.

Unsmoothed, the score for class Y is multiplied by zero, so P(Y | x) = 0 exactly, whatever the other features say. A row with ten strong indicators of Y and this one feature present is classified as X with certainty. That is a veto produced by the absence of one combination in 40 samples.

Smoothed, class Y's score is multiplied by 0.0238 rather than 0 — a heavy penalty, roughly a factor of 42 against it, which is appropriate since the feature genuinely never appeared with Y. But it is now a penalty that other evidence can outweigh. If the remaining features favour Y by more than 42 to 1 in likelihood ratio, Y wins. The difference between smoothing and not is the difference between strong evidence and an unappealable ruling.

Severe overfitting — high variance. Perfect training accuracy with CV near chance-plus-a-bit means the model has memorised 2,000 rows.

(b) increase λ first — it directly reduces variance and costs one training run. (d) collect more data is strictly effective, since variance falls roughly as 1/n, but it is usually the slowest and most expensive option, so it ranks second on merit and last on practicality.

The remaining three all make it worse. (a) more features adds parameters and therefore variance. (c) deeper trees is directly more capacity. (e) reduce k moves k-NN toward its highest-variance setting, and k = 1 guarantees the 1.000 training accuracy this model already has — which is a hint that the model may already be 1-NN, in which case raising k is the whole fix.

The general shape: naming the condition before choosing the fix is the entire exercise, because underfitting reverses this ranking completely.

The experiment. Collect new students — the only thing that adds information here. Roughly 40 to 60 would let a 4-fold stratified cross-validation resolve differences of about 10 percentage points, and the folds would then be large enough for the standard deviation to be interpretable. Score all four models on the same folds with the same preprocessing fitted inside each fold, report mean and spread, and hold out a final test set touched once. Standardise the features for k-NN and the SVM, and use identical folds across models so the comparison is paired.

What to prefer meanwhile. The structural evidence already available says the two linear models are the better bet: the label is defined by a threshold on marks, marks are close to linear in the two features, and a linear boundary therefore matches the generating process. The tree needed a one-row leaf to fit, and k-NN's answer depends on the scaling. So logistic regression is the defensible default, with the SVM as its equal.

If it comes back inconclusive — four means within one standard error of each other — then the honest conclusion is that the data does not distinguish them, and you choose on the other criteria of 2.3.7: prediction cost, interpretability, calibration, and how the model behaves when a label is wrong. That is not a failure of the experiment. Discovering that a difference does not exist is a result, and reporting a winner anyway is the specific dishonesty that 2.1 exists to prevent.