Nobody tells you the answer.

The symmetry is worth noticing: any set of points arranged symmetrically about a centre has that centre as its centroid, and you can read the centroid off by inspection rather than by averaging. Each of the spine's three clusters is built this way, which is why every centroid there is a whole number.

Compute squared distances to (1,2) and (4,8). Group L is plainly nearer S1. For group P: S5 (7,2) gives 36 against 45, so it goes to S1's cluster; S6 (9,2) gives 64 against 61, so it goes to S9's cluster. Check S7 (8,1): 50 against 65, to S1. S8 (8,3): 50 against 41, to S9.

The stable k = 2 answer merges L and P and keeps A separate, because the vertical gap between A and the others is larger than the horizontal gap between L and P. WCSS 84 against 180 for one cluster and 12 for three — and that gap between 84 and 12 is what makes the elbow in section 3.2 so sharp. Note also that at k = 2 the two centroids are (5,2) and (5,8), both sitting at x₁ = 5, so the boundary is horizontal and the practice-hours feature has stopped mattering entirely.

The comparison is meaningless as stated. WCSS decreases monotonically in k for a simple structural reason: any k-cluster solution can be improved by splitting one cluster in two, so the optimum at k+1 is never worse than at k. At k = n, WCSS is exactly zero. So "lower WCSS at higher k" is guaranteed and carries no information about whether the extra clusters are real.

This is the unsupervised twin of 2B's problem with training accuracy: a number that improves automatically as capacity grows cannot be used to select capacity. In 1.6's vocabulary, WCSS is a training loss.

What would settle it. Plot WCSS against k and look for a knee rather than a minimum, and compute silhouette, Davies–Bouldin and Calinski–Harabasz, all of which have interior optima — that is section 3.2. Check stability by clustering bootstrap resamples and asking whether the same groups reappear. And look at the clusters: if two of the eight differ only in ways nobody can act on, they are not clusters worth having, whatever the arithmetic says.

k-means assumes convex, roughly spherical, similarly sized regions. Nearest-centroid assignment produces a Voronoi tessellation: straight boundaries placed halfway between centroid pairs. Squared Euclidean distance makes the regions isotropic, so an elongated cluster gets cut in half rather than recognised.

k-NN assumes only local smoothness. Its boundary is assembled from perpendicular bisectors between individual points rather than centroids, so it can trace any shape given enough data — the jagged outline of 2.3.7's comparison figure. It buys that flexibility with variance and with a complete dependence on feature scaling.

A decision tree assumes axis-aligned rectangles. It cannot draw the diagonal of 2.3.4, but it also does not care about scaling at all, since splits depend only on the ordering of values. Its regions are convex like k-means', but restricted to boxes rather than arbitrary half-space intersections.

The unifying point, and the reason all three appear in this course: each method's failure mode is the shape it cannot represent, and no amount of data fixes it. k-means will never find a crescent, a tree will never find a diagonal, and k-NN will never work in twenty dimensions. Choosing a method is choosing which shapes you are willing to miss — which is 2.3.7's closing point, restated with the labels removed.

The largest gap between consecutive merge heights is from 2 to 4, so the natural cut gives two clusters: {0,1,3} and {7,8}. Under complete linkage the heights would be 1, 1, 3, 8 instead — merge 3 at |3 − 0| = 3 and merge 4 at |8 − 0| = 8 — giving the same two clusters with a more emphatic gap, exactly as in Worked 3.1.2.

First the distances from S13 to its nearest points in each cluster: to S6 (9,2) is 3; to S8 (8,3) is √(1+4) = 2.2361; to S10 (6,8) is √(9+9) = 4.2426, and to S11 (5,7) is √(16+4) = 4.4721; to S1 (1,2) is √(64+9) = 8.5440.

Single linkage. The nine within-cluster merges still happen first at 1.4142. Then the smallest remaining distance is S13 to cluster P at 2.2361, so S13 joins P before anything else merges. The tree then continues: P∪{S13} joins L at 4.0, and A joins at 5.0. So S13 is absorbed into P early and cheaply, and the dendrogram gives no hint that it is unusual.

Complete linkage. S13's distance to the whole of P is its furthest member: to S5 (7,2) is √(4+9) = 3.6056. That is larger than the 2.0 at which P completes, so P forms first and S13 joins at 3.6056 — still before the big merges at 8.0, but visibly later and at its own height. The dendrogram shows a lone leaf attaching high up, which is the visual signature of an outlier.

The lesson is that neither linkage can refuse S13. Hierarchical clustering, like k-means, partitions everything; only DBSCAN in the next section has the option of saying no.

It says the data has two groups and the split between them is extremely clean. Everything below 0.4 merged gradually, then a gap of a factor of seven before the final merge. Cutting anywhere from 0.4 to 2.8 gives the same two clusters, so the two-cluster answer is stable across a very wide range of thresholds — the strongest evidence a dendrogram can offer.

What to check. First, the sizes: a "two-cluster" structure where one cluster has 199 genes and the other has 1 is an outlier, not a grouping, and the dendrogram alone does not distinguish those cases. Second, whether the split tracks a batch or processing artefact rather than biology — if the two clusters correspond to two sequencing runs, the structure is real and uninteresting. Third, the linkage: under single linkage a gap this clean is harder to achieve, so if this is complete or Ward linkage some of the gap may be the method's own bias toward compact balanced clusters.

And below 0.4 there is a second question worth asking. That range of gradual merges may contain sub-structure worth cutting at, and the way to find out is to look for the largest gap within it rather than treating everything below the big split as homogeneous.

Merging two singletons a and b creates a cluster whose centroid is their midpoint, so the new within-cluster scatter is ‖a − m‖² + ‖b − m‖² = ‖a − b‖²/2. Both singletons contributed 0 before, so the increase in WCSS is exactly half the squared distance. Minimising that increase is therefore the same as minimising the squared distance, and Ward's first merge is the closest pair — one of the six diagonal pairs at √2, costing 2/2 = 1 in WCSS.

Contrast the axis pairs at distance 2, which would cost 4/2 = 2. So Ward's opening moves match complete linkage's here, forming the six pairs before joining them.

Why they agree in general. Ward is greedy minimisation of precisely the objective k-means minimises. The two differ only in how they search: k-means iterates from an initialisation and can escape a poor early configuration by reassigning points, while Ward builds bottom-up and can never undo a merge. So Ward inherits k-means' assumptions — compact, spherical, similarly sized clusters — and adds greediness, which is why Ward's result is often used to initialise k-means rather than to replace it. The practical combination is Ward for the dendrogram and the choice of k, then k-means from Ward's centroids to polish the assignment.

Note that 0 and 2 are 2.0 apart, beyond ε, yet they share a cluster because 1 is core and links them — density-connectivity again. Raise minPts to 3 and only the point at 1 stays core, so the cluster becomes {0,1,2} still (0 and 2 become border points) while {6,7} collapses entirely to noise. One notch of minPts destroyed a real cluster.

S13 becomes a border point as soon as it falls within ε of some core point. Its distances to the nearest members of cluster P are: to S8 (8,3) √5 = 2.2361, to S6 (9,2) 3.0000, to S5 (7,2) √13 = 3.6056, to S7 (8,1) √17 = 4.1231.

Two things follow. S13 joins the cluster but cannot extend it, so it contributes nothing further. And its status changed without any change to the data — a parameter moved by 0.74 and a point went from "anomaly worth investigating" to "ordinary member of group P". If the purpose of the analysis were outlier detection, that parameter choice is the analysis, and it should be justified from a k-distance plot rather than left at a default.

62% noise means the parameters are far too strict, or the data has no density structure at that scale. Noise is meant to be a small minority; when it is the majority, DBSCAN is not describing the data.

Three candidate causes, in order of likelihood. Features are unstandardised, so ε is being spent almost entirely on whichever feature has the largest units — check the feature ranges first, it costs nothing. ε is too small or minPts too large, which the k-distance plot will show directly: plot each point's distance to its minPts-th neighbour, sorted, and if the current ε sits well below the knee, that is the answer. Or the dimensionality is too high, which is the serious case — with fifty features, distances concentrate and no ε separates dense from sparse.

What to do about the third case. Do not simply raise ε until the noise count looks acceptable; that is fitting the parameter to the desired conclusion. Reduce dimension first, then re-plot the k-distance curve. And consider whether the density assumption suits the data at all: transaction data often has continuously varying density rather than dense islands in empty space, in which case DBSCAN's single global threshold is the wrong model and HDBSCAN or a mixture model is the better fit.

What noise buys. Robustness of the cluster descriptions. In Worked 3.1.3 the three DBSCAN clusters have centroids exactly at (2,2), (8,2) and (5,8), because S13 was excluded; k-means' third centroid was pulled to (8.2, 2.6) and its scatter tripled. Every summary statistic you compute per cluster is contaminated by forced assignments, and noise labelling prevents that. It also turns outlier detection into a by-product rather than a separate analysis.

What it costs. Coverage. If those 31,000 transactions of P3.1.3.3 are real customers, refusing to describe them is not a result. It also costs the ability to score a new point: k-means assigns a new student by comparing to three centroids, and DBSCAN has no such rule, so deploying it requires re-running on the combined data or fitting a classifier to its output. And it costs determinism at the borders.

When to use which. k-means when you expect compact groups of similar size, need a reusable model, and have enough data that the restarts of 3.1.1 are cheap — it is the default for good reason. Hierarchical when n is small enough for an O(n²) matrix and the nesting is itself informative, as in taxonomy or document organisation, or when you want to see how well-supported each choice of k is before committing. DBSCAN when cluster shapes are unknown or non-convex, when outliers are the interesting part rather than a nuisance, and when the dimension is low enough for density to mean something.

And the meta-point, which is 2.3.7's again: all three ran on the same twelve students and agreed. That agreement is evidence the structure is real, and running more than one method is the cheapest robustness check available in unsupervised learning — because there is no test set to appeal to.

The point is three times closer to the first mean in squared distance and the first component is also more common, so both effects push the same way. Note how to check this kind of answer: if the two means were equidistant, the responsibilities would be exactly the mixing weights (0.6, 0.4), so any deviation from that is the distance term talking.

Full covariance must improve −2 ln L by nearly 15 over spherical just to break even, and with 12 data points and 17 parameters it is very unlikely to earn that honestly — it is more likely to find the singularity of the depth box. On this dataset spherical is not merely adequate, it is the only defensible choice, and BIC says so before you run anything.

The general shape is worth remembering: full covariance costs O(kd²) parameters, so it becomes unusable in high dimensions long before the data does. At d = 50 and k = 5, full covariance needs 6,375 parameters and diagonal needs 504.

That component has collapsed onto a single data point. A weight of 0.004 on n points means an effective count of 0.004n — with 250 points, exactly one — and a near-zero determinant means the covariance has shrunk to a spike. This is the singularity of the depth box, found by EM exactly as predicted. The reported log-likelihood is inflated and meaningless, so any BIC computed from it is also meaningless, and the remaining four components have been fitted to the data with one point removed.

What to do, in order. Add covariance regularization — a small constant on the diagonal — which is one line and usually sufficient. Reduce the covariance type from full to diagonal or tied, which removes most of the freedom the collapse exploited. Reduce k, since a spurious component is often a sign that five is more than the data supports; refit at k = 2, 3, 4 and compare BIC properly. And check whether the point is a genuine outlier or a data error, because either way it deserves attention rather than a dedicated Gaussian.

What not to do is accept the fit because the likelihood is the highest you have seen. Unbounded objectives reward degenerate solutions, and a higher likelihood here is evidence of the pathology rather than against it.

Against Naive Bayes. Both are generative: both model P(class) and P(x | class) and combine them with Bayes' theorem. Gaussian Naive Bayes is a Gaussian mixture with diagonal covariance — that is exactly what the conditional-independence assumption means geometrically, since a diagonal covariance has no off-diagonal terms and so no modelled correlation between features. The single difference is supervision: Naive Bayes is told each training point's class and estimates each class's parameters by direct counting in one pass, while a GMM must infer the classes and so needs EM. So a GMM is unsupervised Naive Bayes, and full covariance is what you gain by dropping the naive assumption.

Against k-means. k-means is EM for a mixture with equal, spherical, fixed covariances and hard assignments, and it is the limit of a GMM as σ² → 0 — Worked 3.1.4's σ² = 1 column already shows the responsibilities collapsing to 0 and 1. Lifting the three restrictions buys three things: elongated and rotated clusters, clusters of different sizes and spreads, and soft assignments that record ambiguity instead of discarding it.

What it costs, and the honest conclusion for this dataset. Parameters, and therefore variance — P3.1.4.2 counted 17 for full covariance against 12 data points. The spine's clusters are spherical, equally sized and well separated, which is precisely the case where k-means' restrictions are true rather than merely convenient, so the GMM's flexibility earns nothing here and BIC penalises it for asking. The value of knowing the relationship is not that GMMs are better; it is that you can now say what k-means assumes, because the assumptions are visible as the special case rather than hidden in the algorithm.

The negative value is the diagnostic. It says the point is on average closer to a different cluster than to its own, so it is very likely misassigned — and note the denominator switches from b to a as soon as a becomes the larger, which keeps s inside [−1, +1]. A silhouette plot with a run of negative bars in one cluster is telling you that cluster should be dissolved or merged.

Both indices are decisively worse at k = 2: CH falls by a factor of 5.5 and DB doubles. The reason is visible in S₁ = 3.0811 — the merged cluster's spread has tripled because it now spans two genuine groups, while the separation to A has not increased at all. That ratio of spread to separation is the whole content of both indices, and merging clusters attacks the numerator.

The disagreement is the finding, and it is a common one. Three indices built from the same two ingredients have picked three different answers, which means the data does not have a single clean cluster structure that all three can see. Do not pick the index whose answer you like.

First, check the mechanics. Were the features standardised? Are there dominant outliers pulling centroids around? Is a categorical variable being treated as numeric? These explain a surprising share of index disagreements and cost nothing to rule out.

Then test stability, which is more informative than any index. Cluster many bootstrap resamples at each candidate k and measure how often the same pairs of customers land together — the adjusted Rand index of section 3.4 applied between two clusterings rather than against truth. A k whose partition reproduces across resamples is real; one that changes every time is an artefact, whatever its silhouette. Add the gap statistic to get a null baseline, since with 40,000 points in high dimensions the indices may all be describing noise.

And then decide on external grounds, which is legitimate. Clustering here is presumably a means to something — targeted campaigns, tiered support, a product roadmap. If the business can act on four segments and not on nine, then k = 4 is the right answer regardless of Calinski–Harabasz, and the honest report says so: "the geometry does not strongly prefer any k; we chose 4 for actionability and verified it is stable under resampling." That is a better answer than a spurious optimum, and it is the shape of most real clustering work.

What is the same. Both are model-selection problems in which a naive objective improves monotonically with capacity — training accuracy in 2.1, WCSS here — so the naive objective cannot be used, and both need either a held-out measurement or an explicit complexity penalty. Calinski–Harabasz's degrees-of-freedom terms and a GMM's BIC are penalty-based; 2.1's validation split is measurement-based.

What is structurally different, and it is decisive. In 2.1 there is a ground truth on the held-out fold, so "did the model get this right" is a well-posed question. Here there is no ground truth at all, so cross-validation has nothing to score. You can hold out points and assign them to the learned clusters, but the only thing you can then measure is how well they fit the clusters you already chose — and a clustering with more clusters will always fit them better, which reproduces exactly the problem you were trying to escape. There is no analogue of "correct" to appeal to.

What is available instead. Three things, in increasing order of value. Internal indices, which measure geometric quality and are biased toward spherical clusters. Stability, which asks whether the same partition reappears on resampled data — this is the closest genuine analogue of cross-validation, since it measures reproducibility rather than correctness. And external validation against labels you happen to have but did not cluster on, which is section 3.4 — the only one of the three that measures anything like correctness, and it is available only when you already have the answer you claimed not to need.

The honest conclusion is that unsupervised model selection is fundamentally weaker than supervised model selection, and no amount of index arithmetic changes that. Which is why the most valuable habit in this unit is running several methods and believing the structure only when they agree — as they did on this dataset, and as they usually will not on yours.

Correlation is 4/√(5 × 5) = 0.8, and the depth box's shortcut gives (1 + 0.8)/2 = 0.9. ✓ With 90% in one component and a clean gap between 9 and 1, this is a case where reducing to one dimension is well supported — the opposite situation to Worked 3.3b, where the eigenvalues were 9.27 and 7.09 with no gap at all.

A zero eigenvalue means the covariance matrix is singular and the data has no spread in the direction (1, −3) — every point lies exactly on a line through the mean with direction (3, 1), so the second feature is a deterministic function of the first, x₂ = x₁/3 plus a constant.

Here reducing to one dimension is lossless: reconstruction error is exactly zero and nothing is given up. This is the ideal case for PCA and it is also a warning sign about the data — a perfectly redundant feature usually means an error in data collection or a derived column that should not have been included. Note the connection to 2.2: this is exact multicollinearity, which made the normal equations singular there and makes the covariance matrix singular here, for the same underlying reason.

The comparison is not valid as evidence, because silhouette is not comparable across spaces of different dimension. Distance concentration means that in 200 dimensions all pairwise distances become similar, so a(i) and b(i) converge and every silhouette is pushed toward 0. A rise from 0.28 to 0.61 is largely a property of the dimension count, and you would see much of it even if PCA had projected onto ten random orthogonal directions. That is the control experiment to run.

That said, PCA may genuinely have helped, for a real reason. The discarded 8% of variance is spread across 190 directions, so those directions are individually almost pure noise, and removing them removes noise from the distance computation. Denoising before clustering is a legitimate and common use of PCA. The point is that the silhouette comparison does not demonstrate it.

What would. Check stability: does the 10-D clustering reproduce across bootstrap resamples more consistently than the 200-D one? Compare both against any labels you have that were not used for clustering, using section 3.4's metrics, which do not depend on dimension. Try 5, 10, 20 and 50 components and see whether the result is stable across that range or an artefact of choosing 10. And run the random-projection control, which is one line of code and settles the main question.

Finally, keep Worked 3.3b in view: 92% retained sounds safe and says nothing about whether the discarded 8% contained the separating direction. Check the scree plot for a genuine gap at 10.

All three are answering the question "what should I do with a direction in feature space that carries little information", and they give three different answers to it.

PCA discards it outright. Directions are ranked by variance and the tail is deleted, so a low-variance direction contributes nothing to the projection. The decision is made without reference to any task, which is why Worked 3.3b went wrong — low variance was mistaken for low value.

Ridge regression shrinks it. Section 2.2 showed that adding λI to the scatter matrix fixes singularity, and the mechanism is exactly this: a direction with small scatter has its coefficient shrunk hard, because λ is large relative to that direction's own eigenvalue. So ridge does not delete the direction, it distrusts it in proportion to how little data supports it. That is strictly gentler than PCA's cut, and it is why "principal component regression" — PCA then least squares — is generally worse than ridge: it makes a hard yes/no decision where a soft one is available.

k-means is blind to it. Squared Euclidean distance weights every direction equally regardless of variance, so a low-variance direction contributes to the clustering exactly as much as a high-variance one per unit of difference. This is why standardisation matters so much for k-means and why the scaling trap of 2.3.2 recurs here: k-means has no mechanism at all for down-weighting an uninformative feature, and you must do it yourself.

The unifying principle, which is 1.7's: every one of these is a statement about which directions to trust, and the only difference is whether the statement is hard, soft, or absent. Ridge's soft version is usually the best-behaved, which is a general lesson worth carrying into Unit 4, where weight decay is ridge under another name.

Six of the eight points are "correctly" placed, which sounds like 75% agreement, and the ARI is 0.125. The gap is the chance correction doing its job: with two clusters and two classes of equal size, a random assignment already gets most pairs right, so 6 out of 8 is barely above the baseline of 5.14. This is exactly why the adjustment exists.

Twelve singleton clusters. Each cluster contains one student, so knowing the cluster tells you the outcome exactly: H(C|K) = 0, so h = 1 − 0/H(C) = 1.0000. But each class of six is split across six clusters, so H(K|C) is large. With H(K) = ln 12 = 2.4849 and H(K|C) = ln 6 = 1.7918, completeness is 1 − 1.7918/2.4849 = 0.2789. V-measure = 2(1)(0.2789)/1.2789 = 0.4362.

One cluster of twelve. Knowing the cluster tells you nothing, so H(C|K) = H(C) and h = 0.0000. Completeness needs care here: with a single cluster H(K) = 0 and H(K|C) = 0, so the ratio H(K|C)/H(K) is 0/0 and the formula gives nothing. The convention is to define completeness as 1.0000 in this case, on the grounds that no class has been split — and libraries implement that convention rather than deriving it. V-measure is then 2(0)(1)/(0+1) = 0.0000.

Watch for that 0/0 whenever a degenerate clustering appears. The same thing happens to homogeneity if every point shares one class, since then H(C) = 0. These are not edge cases invented for exams — a clustering that collapses to one cluster is a common real failure, and a metric that returns nan rather than a number is often the first sign of it.

These are the two degenerate extremes, and they show why the pair is needed. Each measure alone can be driven to 1 by a clustering that is obviously worthless, and only their combination resists both. That is precisely the relationship precision and recall have in 2.4.2 — and it is why V-measure is a harmonic rather than arithmetic mean, since the harmonic mean is 0 whenever either component is 0.

There is no contradiction: the two numbers measure different things and both are probably right. Silhouette 0.68 says the four groups are geometrically clean — tight and well separated in feature space. ARI 0.04 says those groups have almost nothing to do with the diagnosis. Both can be true simultaneously, and here they almost certainly are.

The most likely explanation is that the clustering has found real structure that is not the diagnosis. Patient records contain strong, clean structure along axes such as age, sex, site of care, referral route or which instrument recorded the measurements — and any of those can dominate the geometry while being irrelevant to disease. A clustering that separates patients by scanner model will look magnificent on silhouette and score zero against diagnosis. The first thing to do is characterise the four clusters: compute the mean of every feature in each, and the identity of the structure will usually be obvious.

What follows depends on the goal. If the aim was to discover diagnostic subtypes, this is a negative result honestly obtained, and the response is either better features or a supervised model, since the labels exist. If the aim was to find any structure, then discovering that scanner model dominates the variance is a valuable finding — it is a confound that will damage any downstream model, and it was found for free.

What not to do is tune k to raise the ARI. With labels in hand, maximising an external metric is supervised learning, and the resulting score is a training score. If the diagnosis is the target, use Unit 2.

Homogeneity and completeness are precision and recall, generalised. Precision asks what fraction of the points a model put in the positive class actually belong there — a per-prediction purity. Homogeneity asks what fraction of the information in a cluster's membership is accounted for by its class composition — a per-cluster purity. Recall asks what fraction of the actual positives were found; completeness asks whether a class's members ended up together. Both pairs have the same two degenerate extremes: predicting positive for one certain case maximises precision, and a singleton clustering maximises homogeneity; predicting positive for everything maximises recall, and one big cluster maximises completeness. Both pairs are combined by a harmonic mean — F1 and V-measure — for the same reason, that the harmonic mean refuses to reward either extreme.

ARI is Cohen's κ, applied to pairs instead of points. Both have the identical form (observed − expected) / (maximum − expected). Both exist because the raw agreement rate has a large baseline that depends on the marginal distributions rather than on the quality of the agreement. Both are 0 at chance, 1 at perfection, and negative when performance is worse than chance. The only difference is the unit of agreement: κ counts points on which two labellings agree, ARI counts pairs of points on which two partitions agree about togetherness — and ARI must use pairs precisely because cluster names are arbitrary, whereas κ can use points because class names are not.

The shared structure, stated once. Every honest evaluation metric in this course does one of two things: it subtracts a baseline so that 0 means "no skill", or it combines two quantities that can each be gamed alone. Accuracy does neither, which is why it needed the majority-class rate quoted beside it in 2.4.2; the Rand index does neither, which is why it needed adjusting. Once you see the pattern you can predict what a new metric must look like before you read its definition.

minPts: the common heuristic is 2d = 16, with d + 1 = 9 as the floor. Start at 16 and lower it if too much is labelled noise.

ε: do not guess. Standardise all 8 features first, since a single ε applies to all of them jointly. Then for every point compute the distance to its 16th nearest neighbour, sort those distances descending, and plot. The curve will be flat over most of its length and rise sharply at the left-hand end where the noise points sit; the knee is the value of ε that separates dense from sparse, and that is the starting point.

Then check the result rather than accepting it: what fraction is noise, how many clusters, and are they of plausible sizes? And with 8 features, be alert to the concentration problem of 3.1.3's depth box — if the k-distance curve has no knee, that is the symptom, and reducing dimension first is the response.

What to expect here. k = 3 should be extremely stable, close to ARI 1.0, because the three clusters are separated by gaps of 6 and 6.7 against within-cluster distances of at most 2 — no resample that retains a few members of each group will fail to find them. k = 2 should also be stable, reliably merging L and P, since that merge is forced by geometry. k = 4 and k = 5 should be visibly unstable: the fourth cluster is made by splitting a diamond, and which diamond gets split and along which diagonal will vary from resample to resample.

So stability should select 3 — agreeing with all four indices in 3.2. Two caveats worth stating. With n = 12 a bootstrap resample contains only about 8 distinct students, so the estimates will be noisy and the whole exercise is a demonstration rather than evidence. And stability would also endorse k = 2, which is not wrong so much as coarser — stability tells you which answers are reproducible, not which is best, so it belongs alongside the indices rather than instead of them.

Three problems, in increasing order of seriousness.

The number is unbaselined. A silhouette of 0.42 is around what uniformly random points in a box score at moderate k. Without a gap statistic or a comparison against a null model, 0.42 is not evidence of structure — it may be evidence that k-means partitions anything. That single number is doing all the work in this report and it cannot bear the weight.

No stability or corroboration is reported. Nothing tells us whether the same seven groups appear on a resample, whether other values of k scored nearly as well, or whether another method agrees. If silhouette at k = 6 were 0.41, the choice of 7 is noise. Report the whole curve, not its argmax.

And "represent seven distinct personas" is not something the analysis can support. This is the deepest issue. k-means partitions any dataset into seven convex regions whether or not seven groups exist; producing seven clusters is what the algorithm does, not a finding. Calling them personas asserts that they are meaningful, coherent and actionable, and none of those was tested. Also missing: whether features were standardised, how many restarts were used, and what the seven clusters actually differ on — without cluster profiles the reader cannot judge any of it.

A defensible version. "Features standardised; k-means with 50 restarts. Silhouette across k = 2 to 12 peaked broadly at 6 to 8 with values 0.39 to 0.42, and the gap statistic favoured 3. Bootstrap stability was high for k = 3 (mean ARI 0.91) and moderate for k = 7 (0.62). We report k = 7 for operational reasons, with cluster profiles in Appendix B, and note that the geometric evidence for seven groups specifically is weak."

Transfers unchanged. Grid and random search over the hyperparameter space; the discipline of not letting a final reported number come from data used to choose the setting; and the practice of reporting the whole search curve rather than only its optimum. All of these are about procedure rather than about labels, so they survive intact.

Has an analogue. Cross-validation becomes stability analysis. Both resample the data and ask whether the conclusion survives, but they measure different things: cross-validation measures expected performance against truth, stability measures reproducibility of a partition. Regularization becomes an information criterion — BIC's p ln n penalty plays the role that ridge's λ‖w‖² played in 2.2, penalising complexity, with the difference that BIC's penalty is derived rather than tuned. And the majority-class baseline of 2.4.2 becomes the gap statistic's null reference, serving the identical purpose: telling you what a useless answer would score.

Simply unavailable. A held-out performance estimate, because there is no performance to estimate. A learning curve against training-set size, since there is no error to plot. Early stopping on a validation metric. And most consequentially, the guarantee that a better validation score means a better model — there is no quantity here that plays the role validation accuracy plays there.

The one-sentence version. Supervised tuning asks "which setting predicts held-out labels best" and gets a real answer; unsupervised tuning asks "which setting produces a partition that is geometrically clean, reproducible, and better than what random data would give" — three weaker questions that together are the best available substitute, and a substitute is what they remain.

Both seedings converge to the same answer, which is the split along the longer dimension — the rectangle is 4 wide and 3 tall, so cutting vertically leaves less residual scatter. Check the alternative: splitting horizontally into {(0,0),(4,0)} and {(0,3),(4,3)} gives centroids (2,0) and (2,3) with each point 2 from its centroid, so WCSS = 16. Worse, as expected.

The first value is weakly positive: the point is on a boundary. It is in the right cluster, but only just, and its own cluster is barely tighter around it than the neighbouring one. A silhouette plot full of bars near 0.14 says the clusters are touching rather than separated.

The second is clearly negative and means the point is closer on average to another cluster than to its own — almost certainly misassigned. Note that the denominator switched from b to a, which is what keeps s within [−1, +1]. Note also that the far cluster at 9.0 never entered either calculation: silhouette only ever looks at the nearest rival.

Raising minPts by one destroyed a real cluster. {10,11} is a genuine pair, well separated from everything else, and it vanished because two points can never satisfy a three-point density requirement. This is Worked 3.1.3b's failure at minPts = 4 in miniature, and it is why minPts should be set from the dimensionality rather than nudged until the output looks nice.

Note also that 0 and 2 changed status from core to border without moving. Their cluster membership survived only because 1 remained core and could reach them.

Cross-check with the shortcut: correlation is 6/√(10 × 10) = 0.6, and (1 + 0.6)/2 = 0.8. ✓ With 80% in one component and eigenvalues of 16 against 4, this is a case where reducing to one dimension is reasonable — unlike the spine, where 9.27 against 7.09 offered no gap at all and reduction destroyed the clusters.

The single-cluster case is the sanity check every one of these metrics must pass, and ARI passes it exactly: the observed pair count equals the expected pair count, so the numerator is precisely zero. A metric that gave partial credit for putting everything in one bucket would be useless, which is exactly the criticism of the unadjusted Rand index — it scores this table 20/45 = 0.4444.

With equal weights the decision boundary is the perpendicular bisector regardless of spread, which is exactly k-means' boundary — another way of seeing that k-means is a GMM with equal weights and shared spherical covariance. Unequal weights shift the boundary toward the less common component, and the shift grows with σ², because a larger spread makes the distance evidence weaker relative to the prior. That is Bayes' theorem behaving exactly as 2.3.6 said it would.

The two criteria disagree, and the disagreement is informative rather than a tie to be broken. Silhouette says the two-cluster partition is geometrically cleaner. Stability says the five-cluster partition is the one that reproduces — resample the data and you get the same five groups 93% of the time, whereas the two-group answer changes substantially between resamples.

The most likely explanation is hierarchical structure. The data has five real groups arranged in two loose super-groups. Silhouette prefers k = 2 because the super-groups are further apart, so b(i) is large; stability prefers k = 5 because the five groups are the reproducible unit while the boundary between super-groups is arbitrary and moves with the sample. A stability of 0.55 at k = 2 is genuinely poor and is the stronger signal here, since a clustering that does not survive resampling is not describing the population.

What to report. Both, with the structure made explicit: run hierarchical clustering to check whether the five groups nest inside the two, and if they do, report the hierarchy rather than a single k. If forced to one number, k = 5 — reproducibility is the more demanding test, and 0.44 is a perfectly ordinary silhouette for five clusters while 0.55 stability is a red flag. And add the gap statistic, which will say whether either beats a null model at all.

Why they differ. k-means minimises squared Euclidean distance, which sums every feature's contribution. Projecting onto two components changes those distances: any difference lying in the discarded directions vanishes entirely, so two points far apart in the full space can be adjacent after projection. Since the distances change, the partition minimising WCSS changes too.

Where it helps. High dimension with most directions carrying noise. With 500 features of which 10 are informative, the 490 noise directions each add a small random amount to every pairwise distance; summed, they can swamp the real signal, and this is the concentration problem of 3.1.3's depth box. Projecting onto the top components removes most of that noise, and the clustering improves genuinely rather than as an artefact.

Where it hurts. Worked 3.3b, exactly. The discarded direction was the one separating two of the three clusters, so PCA merged them before k-means ever ran — and it did so while retaining 56.67% of the variance, which by its own criterion was the optimal choice. The danger is that this failure is silent: the clustering that follows looks fine, and nothing in the output says a cluster was destroyed upstream.

The distinguishing question is whether the discarded variance is noise or structure, and variance alone cannot tell you. Check the scree plot for a genuine gap; if the eigenvalues decline smoothly there is no safe cut. And cluster both ways — if the answers agree, the reduction was harmless.

Both agree on three, which is reassuring — agreement between linkages is a genuine robustness check, since the two methods have opposite biases.

Complete linkage supports it more strongly, and in two distinct ways. Its decisive gap is 5.1 against single's 3.4 in absolute terms, and more tellingly the ratio of the first between-cluster merge to the last within-cluster merge is 7.0/1.9 = 3.7 for complete against 4.2/0.8 = 5.25 for single. So on the ratio measure single linkage looks better, which is a reminder to say which measure you are using. The stronger argument for complete here is its final merge at 9.5 against 7.0: it also separates the top two clusters decisively, whereas single linkage's last two merges at 4.2 and 4.4 are nearly identical, meaning the three-cluster and two-cluster solutions are almost equally supported. That near-tie is exactly the chaining signature of 3.1.2.

The clustering is technically sound and that is not the relevant question. Silhouette 0.58 and stability 0.88 say the five groups are real and reproducible. Neither number says anything about whether routing applicants by them is acceptable, and the ethnic concentration means it very likely is not.

The mechanism is 2.6's, and clustering does not escape it. Nobody fed ethnicity to the algorithm, and it made no difference — features like postcode, school, previous employer and phrasing are all correlated with it, so the geometry reproduces the demographic structure without ever naming it. That is fairness through unawareness failing exactly as 2.6 said it would, and if that cluster is routed differently, applicants receive different treatment by ethnicity through a proxy.

What to do. Audit the routing outcomes by group using 2.6's metrics — selection rate, and if any downstream outcome is observable, recall and precision by group. Identify which features drive the concentrated cluster, since the answer is usually one or two proxies that can be removed or coarsened. Ask whether clustering is the right tool at all: if the goal is to route applications well, that is a supervised problem with a measurable outcome and Unit 2's machinery, including its fairness constraints, applies. And check the legal position, because in many jurisdictions this is disparate impact regardless of intent.

What not to do is treat the internal metrics as a defence. "Silhouette 0.58, stability 0.88" describes geometry. The applicants are people, the clusters determine what happens to them, and no unsupervised metric in this unit is capable of noticing that.