A model is a claim. This is the audit.

Note the shape of the result: recall 0.80 with precision only 0.62 says the model casts a wide net, flagging 65 to catch 40 of the 50 real positives. Whether that is right depends on the cost of the 25 false alarms.

Accuracy, balanced accuracy and MCC are symmetric: they treat the two classes alike, so relabelling cannot change them. Precision and recall are not, and they swap partners — recall becomes the old specificity and vice versa. Precision changes most dramatically, from 0.6154 to 0.9259, purely by renaming.

The practical consequence is that F1, being built from precision and recall, is also asymmetric, so an F1 score is meaningless without knowing which class was positive. MCC has no such ambiguity, which is the main argument for reporting it alongside.

The least charitable interpretation is that the model never predicts a defect. Always answering "no defect" scores 98% accuracy on a 2% defect rate — so 97% is actually worse than the trivial baseline, and consistent with a model that finds nothing while occasionally raising a false alarm.

The four numbers to request are the raw cell counts TP, FP, FN and TN. Everything else is derivable from them, and unlike any ratio they cannot hide a degenerate model: if TP is 0, the matrix says so immediately.

Two further requests worth making. The majority-class baseline accuracy, so the 97% has something to be compared against. And the cost of a missed defect versus a false alarm, because in manufacturing those differ by orders of magnitude and the threshold should have been set from them rather than left at 0.5. Until you have the counts, "97% accuracy" carries almost no information.

Accuracy responds only to which side of the threshold each probability fell on. Lowering the threshold past one student's score changes exactly one prediction, converting one false negative into a true positive or one true negative into a false positive. Between 0.70 and 0.50 the model gained one TP and one FP, so TP + TN went from 7 to 7 — one gained, one lost. Accuracy counts FP + FN without weighting, so a one-for-one swap is invisible to it. That happened twice in a row here, which is why the figure repeats.

Log loss does not see the threshold at all. It is computed from the probabilities themselves, −Σ[y ln p + (1−y)ln(1−p)]/n, and the threshold appears nowhere in that expression. Moving it cannot change log loss by any amount. This is the same fact the weight-scale control in 2.3.1's widget demonstrated from the other direction: scaling the weights changed log loss while leaving accuracy and the boundary fixed.

So the two metrics are sensitive to orthogonal things. Log loss grades the model — how good are the probabilities. Accuracy, precision, recall and F1 grade the model plus a threshold — a deployment decision. Reporting only accuracy conflates two choices and makes it impossible to tell whether a poor result came from a bad model or a badly chosen cut-off. Report a threshold-free measure (log loss, AUC) for the model and a thresholded matrix for the deployment.

The spread from 0.3462 to 0.6429 is wide because precision and recall are far apart, and each F_β sits nearer whichever it favours. Note also that the arithmetic mean would be 0.600, comfortably above F1's 0.4500 — the harmonic mean is refusing to let recall of 0.90 excuse precision of 0.30.

This profile — high recall, low precision — describes a screening tool. It is the right shape when a miss is costly and a false alarm triggers a cheap second check, and the wrong shape when the flag itself does harm.

Operationally, precision 0.1307 means an investigator opening flagged cases finds fraud in roughly 1 of every 8. At 1,148 flags that is 1,148 investigations to catch 150 frauds, and whether that is acceptable is an economic question about the cost of an investigation against the value of a caught fraud.

Note that accuracy of 0.98952 is below the do-nothing baseline of 0.99800, which is the depth box's pattern exactly. A specificity of 0.99 sounds excellent and produces 998 false alarms, because 1% of 99,800 is a large number. When negatives are abundant, a specificity of 0.99 is not nearly good enough, and this is why fraud systems are tuned on precision at a fixed alert volume rather than on any accuracy-like measure.

Model B, and not by a small margin. In triage a false negative is a high-risk patient sent home, and a false positive is a low-risk patient who waits for a doctor they did not need. Those costs differ by orders of magnitude. Model A misses 60% of high-risk patients, which is not a triage system.

Quantitatively, with β = 3 to say a miss is nine times worse than a false alarm: F₃(A) = 10(0.85)(0.40)/(9(0.85) + 0.40) = 3.40/8.05 = 0.4224 and F₃(B) = 10(0.45)(0.95)/(9(0.45) + 0.95) = 4.275/5.00 = 0.8550. B wins by a factor of two. Choosing β is where the clinical judgement enters, and it should be recorded.

What would change my mind. If a positive flag consumes a genuinely scarce resource — the only available surgeon, a single ICU bed — then B's precision of 0.45 means over half that resource is spent on patients who did not need it, and the false positives start causing harm to third parties. Also, if these are two thresholds on the same underlying model, the right response is neither: get the ROC curve of 2.4.3, and pick the operating point from an explicit cost ratio rather than from two arbitrary candidates.

Three separate problems, and each alone would be enough.

The sample is four positives. Recall 1.0000 means 4 of 4, and the 95% confidence interval for a proportion of 4/4 reaches down to about 0.40. The point estimate is 1.0 and the evidence is compatible with a true recall of one half. Any recall computed from four cases is a number with almost no precision, in the statistical sense.

The threshold was chosen by looking at these ten students. Per 2.1, a split that has influenced a decision cannot report an honest number. Sweeping the threshold to find the best recall and then quoting that recall is selection on the test set, and the quoted figure is optimistically biased by exactly the amount the search bought. The threshold should be fixed on validation data and then applied once.

Recall 1.0000 is trivially available. Set t = 0 and every classifier on earth achieves it. Recall alone is not a measure of skill; the pair (recall, precision) is, and at t = 0.30 precision fell to 0.5714. In 1.6's terms, driving one metric to its extreme on a small sample is fitting the evaluation rather than the data — the same failure as the degree-4 polynomial, relocated from the model to the report.

Cross-check by area: right-steps of width 1/3 occur at TPR heights 1/3, 1, 1 → (1/3)(1/3 + 1 + 1) = (1/3)(7/3) = 7/9. ✓ The two discordant pairs both involve the failure at rank 2, which outranks the passes at ranks 3 and 4.

One swap between adjacent ranks moved AUC by 1/24 = 0.0417 and average precision by 0.0625. That sensitivity is the honest scale of a metric computed from 24 pairs, and it is why an AUC quoted to three decimal places on a ten-row test set is false precision. Note also that the remaining three discordant pairs all involve S7, the pass at rank 7 — that single badly-ranked student now accounts for every error the model makes.

Model B, on this evidence. At 3% prevalence the positive class is rare and is presumably the one you care about, so the depth box's rule says trust the PR summary. Average precision 0.51 against a baseline of 0.03 is a seventeen-fold improvement over chance; 0.34 is an eleven-fold improvement. B is substantially the better model where it matters.

What the disagreement tells you is where each model's errors sit. AUC integrates over all thresholds including very permissive ones, and it normalises false positives by the huge negative pool. A has the better global ranking but is making its mistakes at the top of the list — a few high-scoring negatives, which barely dent FPR but wreck precision exactly where you operate. B ranks slightly worse overall while keeping the top of its list clean.

What to do. Compare them at a fixed operating point you would actually deploy: precision at the top 100 alerts, or recall at a fixed alert budget. That is the number the business cares about and neither summary reports it directly. Also check whether the curves cross, since two models with different error profiles usually do, and the crossing point tells you which model wins in which regime.

Multiplying all weights and the intercept by 4 maps each score z to 4z, and the sigmoid is monotone increasing, so each probability σ(z) maps to σ(4z) — a strictly increasing transformation. The ordering of the ten students is therefore identical. AUC depends only on the ordering, so it is exactly unchanged. Average precision, being also purely ordinal, is unchanged too.

Accuracy at a fixed threshold can change, because the probabilities move even though their order does not. A student at p = 0.54 has z = 0.1602, and after scaling z = 0.6408 gives p = 0.6549 — still above 0.5, so in this particular case no prediction flips. But a threshold of 0.6 would have excluded that student before scaling and included them after. Accuracy grades the model plus the threshold; scaling changes the relationship between the two.

What AUC cannot detect is miscalibration. The 4×-scaled model is wildly overconfident — it will report probabilities near 0.99 for students it has seen once — and its AUC is identical to the well-calibrated version's. A model can have AUC 1.0000 and probabilities that are all wrong in magnitude. If the number will be consumed as a probability, in an expected-cost calculation or shown to a person, AUC is not enough and section 2.4.6 is the missing check.

Decline whenever the estimated probability of default exceeds 11.1%. Note the cost ratio is 8:1 and the threshold is 1/9, not 1/8 — the formula is C₄ₚ/(C₄ₚ+C₄ₙ), not C₄ₚ/C₄ₙ. And note how far this is from 0.5: using the default would approve a great many loans with a 30% or 40% chance of default.

The constraint and the cost-optimal threshold are incompatible, so something must give, and naming which is the actual decision.

Accept a higher expected cost. Raise the threshold until 70% are approved and compute the resulting expected cost. That number is the price of the commercial constraint, and it should be stated to whoever imposed it. Often it turns out to be larger than they expected.

Improve the model instead. The conflict exists because the model's ranking is not sharp enough to find 70% of applicants who are genuinely low-risk. A better model — more features, better calibration — moves the whole curve and can satisfy both. This is the only option that does not involve a trade.

Re-examine the costs. ₹200,000 per default may assume no recovery; if collateral recovers half, the ratio changes and t* rises to 25/125 = 0.20, which may approve close to 70% on its own. Cost estimates are usually the least examined input to the whole calculation.

Change the action space. The choice need not be binary. A middle band could receive a smaller loan, a higher rate, or a manual review, which converts an expensive error into a cheaper one. Two thresholds instead of one is frequently the right answer, and the cost formula extends to it directly.

Tuning on the test set. The threshold is a hyperparameter, so the test set has been used to make a decision and 0.81 is a validation figure. The honest number is whatever F1 that threshold achieves on data not involved in choosing it, and it will be lower. With ten thresholds tried the optimism is modest; with a fine sweep on a small set it can be several points.

Maximising F1 without justifying it. F1 weights precision and recall equally, which is a claim that the two error types cost the same. If that is true, say so and say why. If it is not, F1 was the wrong objective and the threshold is wrong regardless of how it was tuned. F1 also ignores true negatives entirely, which may or may not be acceptable here.

Reporting one number. F1 = 0.81 does not say what precision and recall were, so it does not say what the model does. It also does not say the prevalence, without which F1 cannot be compared to anything.

What to report instead. The threshold-free summary (AUC or average precision, with the prevalence), the chosen threshold and the reasoning behind it, the full confusion matrix at that threshold on a clean test set, and the metrics derived from it. That is four lines rather than one, and it is what a reader needs to judge the claim.

AUC is the area under the whole ROC curve, and many different curve shapes enclose the same area. One model might rise steeply at low FPR and then flatten; another might rise gently throughout. Their AUCs are equal and their curves cross.

The cost-optimal operating point is a single location on the curve, determined by the cost ratio. Formally, the optimal point is where the ROC curve's slope equals (C₄ₚ/C₄ₙ) × (n₋/n₊) — so the cost ratio and the prevalence together select a slope, and you move along the curve until you find it. Two curves of equal area have different slopes at every FPR, so they yield different operating points and different expected costs. The model with the steeper initial rise wins under a high false-positive cost; the other wins under a high miss cost.

The implication is that model selection cannot be separated from the deployment decision. Ranking candidate models by AUC and then choosing a threshold is two steps in the wrong order, because the first step discards the information the second needs. The correct procedure is to fix the cost ratio first, then compare models at their respective cost-optimal points on validation data, then confirm once on the test set. This is the same discipline 2.1 demanded of hyperparameters, applied to the metric itself — and it is why 2.3.7 insisted that the protocol matters more than the model.

RMSE 2.1213 against MAE 2.0000 is a narrow gap, so the errors are fairly evenly sized — consistent with residuals of 1, 2, 2 and 3. Note that MAPE's 10.7% is driven mostly by the first point, where an error of 2 on a value of 10 is 20%; the same error on the value of 34 would be under 6%. That asymmetry across the range is MAPE's characteristic weakness.

The ratio is 180/62 = 2.90, which is very large. Since RMSE equals MAE only when all errors are the same size and grows relative to MAE as errors become uneven, a ratio near 3 says the error distribution is dominated by a small number of very large misses while most predictions are good. An R² of 0.88 is consistent with this: the bulk of the variance is captured and a handful of houses are badly wrong.

What to do. Find them. Sort by absolute residual and look at the worst twenty. The likely candidates are luxury properties outside the training range, where a linear model must extrapolate; data errors such as a misplaced decimal in the floor area; or a genuinely distinct segment — commercial conversions, disputed titles — that the features do not describe.

Then decide what the metric should be. If a single ₹2,000,000 error is catastrophic for the business, RMSE is the right objective and those cases need fixing. If the model is used for typical valuations and the rare mansion is handled manually, MAE or a Huber loss from 1.3 is the better objective, and the ₹62,000 MAE is the honest headline figure. Reporting only RMSE would make a good model look poor; reporting only MAE would hide a real risk.

The 0.8 bin is overconfident — it claimed 80% and delivered 60%. The 0.3 bin is underconfident — it claimed 30% and delivered 40%. Both errors push predictions toward the extremes, which is the classic signature of a model trained without regularization, or of Naive Bayes double-counting correlated evidence. A monotone recalibration that pulls both bins toward the centre would fix both at once, and it would leave the ranking, and therefore the AUC, exactly as it is.

How large is the test set? A 0.6-point gain means about 6 extra correct predictions per thousand. On a 500-row test set it is three cases and could be a single fold's luck. The p-value is only as meaningful as the experiment behind it.

Was the comparison paired, and were the folds shared? A paired test on shared folds is the right instrument, but its p-values are anti-conservative because CV training sets overlap, so p = 0.04 from five folds is weaker evidence than p = 0.04 from independent samples.

How many models were compared to find this one? If twenty variants were tried, one of them clearing p = 0.05 is expected by chance. This is 2.1's selection problem applied to significance tests.

What is the baseline and the prevalence? At 0.912 accuracy, a majority-class baseline of 0.90 would mean both models are barely above trivial, and the entire comparison concerns a sliver of the data. Ask for the confusion matrices.

And what does it cost? Six correct predictions per thousand may be worth a great deal or nothing at all, depending on what the errors cost. Significance is a statement about noise, not about value.

Raising the bar from 0.9 to 0.95 moved the requirement from s ≥ 11 to s ≥ 12, and made the negative side essentially unreachable since s = 0 means a student who did nothing at all. Each additional nine of confidence costs about ln(10)/0.5 = 4.6 units of s here, so with weights this small a high confidence threshold rejects almost everything. The relationship between weight magnitude and how much data clears a confidence bar is worth noticing: it is why overconfident models pseudo-label freely.

No, weight them down. At full weight, 15% of the training signal is actively false and the real labels are outvoted three to two. The three wrong pseudo-labels are also not randomly distributed — they will be clustered in whatever region the initial model misunderstands, so their local influence is far greater than 15%.

A weight of 0.25 keeps the real labels dominant while still letting the unlabelled data shape the boundary. The number itself should be chosen by validation on held-out real labels, like any hyperparameter, and if no weight produces an improvement then this dataset does not satisfy the assumptions and self-training should be abandoned. A 3-in-12 error rate on the most confident predictions is already a warning sign that the model's confidence is not calibrated.

Nothing has been measured. The pseudo-labels were produced by the model, so measuring the model against them measures self-consistency, not correctness. After retraining on its own outputs the model necessarily agrees with them, and that agreement approaches 1.0 whether or not a single label is right. The curve they are celebrating is a tautology.

What has probably happened underneath is the confirmation-bias loop. Round 1's confident errors entered the training set, subsequent rounds gained support for them, and by round 10 the model is emphatically wrong in whatever regions it started out wrong. True accuracy may well have fallen from its round-0 value.

What to do. Set aside a labelled test set before starting and evaluate on it after every round; that curve is the only one that means anything, and self-training should be stopped at its peak rather than at convergence. Also compare against the round-0 model trained on labelled data alone, because if self-training cannot beat that baseline it has no justification. And inspect the pseudo-labels by hand — a sample of thirty is enough to reveal a systematically mislabelled region.

It depends on the geometry of the feature space, which supervised learning does not. The cluster and smoothness assumptions are both statements about distance, and 2.3.2 established that distance in raw feature units is essentially arbitrary — recording practice in minutes rather than hours reordered every neighbour and flipped k-NN's prediction. A label-propagation graph or a consistency-regularization neighbourhood built on unscaled features encodes the same arbitrary geometry, and it will propagate labels along whichever axis happens to have the largest numbers. Supervised learning with a linear model is immune to this, because a coefficient simply absorbs the rescaling.

It also depends on confidence being meaningful, which 2.4.6 showed is not automatic. Self-training's threshold assumes that "0.95" means a 95% chance of being right. For an uncalibrated model — a boosted tree, a Naive Bayes classifier, a network trained to convergence — it may mean 70%, in which case a threshold of 0.95 admits errors at three times the intended rate and the safeguard is doing nothing.

The one check. Hold out labelled data and measure on it after every round. It catches confirmation bias, it catches a violated cluster assumption, it catches miscalibrated confidence, and it catches the case where the whole approach was inapplicable — because all four of those failures show up as true accuracy that fails to improve or actively declines. This is 2.1's discipline with no modification at all: the only number you can trust is the one measured on data that did not influence the model.

Demographic parity: violated badly — X is flagged at two and a half times Y's rate. Equal opportunity: violated — X's recall is 0.75 against Y's 0.50, so qualified members of Y are half again as likely to be missed. Equalised odds: violated on both components. Predictive parity: violated, but in the opposite direction — a flag for group Y is more reliable (0.75) than a flag for group X (0.60).

That reversal is the diagnostic worth noticing. The model is conservative with group Y: it flags rarely, so the flags it does raise are trustworthy, and it misses many qualified people. Closing the recall gap by lowering Y's threshold would flag more of group Y and reduce their precision below X's — the same trade as Worked 2.6b. Whether that is an improvement depends entirely on whether being missed or being wrongly flagged is the greater harm here.

So matching X's precision requires doubling group Y's false positive rate, from 0.0714 to 0.1429 — making the model deliberately worse on group Y in order to make the two groups' precisions agree. That is a real option and it is what predictive parity demands here, and it illustrates why fairness constraints are not free: every one of them is enforced by degrading some metric somewhere.

The alternative reading is that precision parity is the wrong target on this problem. If the harm is being missed, equal opportunity is what matters, and the fix is to raise group Y's recall by lowering its threshold — which also raises its FPR, and lowers its precision below X's. Every route out of this involves choosing which group absorbs which error.

Equal accuracy is nearly uninformative here, and it is not one of the five criteria for a reason. Accuracy pools true positives with true negatives, so two groups can have identical accuracy with completely different error profiles. Worked 2.6a's numbers make the point: had group U's accuracy happened to equal group P's, the recall gap of 0.30 would still have been there. Ask for the disaggregated confusion matrices, which is four numbers per group.

Accuracy also flatters whichever group has the more extreme base rate, per 2.4.2's depth box. A group with few qualified applicants gets high accuracy from rejecting nearly everyone, so equal accuracy across groups with different base rates may indicate that the model is doing something quite different in each.

What to request. Selection rate, TPR, FPR and PPV per group, with counts so the intervals can be judged. The intended fairness criterion, stated in advance, with the reasoning. Whether the label Y is "was hired" or "was successful in the role" — because the first records past hiring decisions and optimising against it reproduces them, which is the untrustworthy-label case. And a model card documenting which populations the evaluation covered, since a group that was too small to evaluate is a group with no evidence at all, not a group with no problem.

Whose costs. The formula t* = C₄ₚ/(C₄ₚ + C₄ₙ) takes two numbers and returns a threshold, and it is silent about where the numbers came from. In practice they are the operator's costs: the bank's loss on a default, the hospital's cost of a follow-up test, the university's cost of a tutoring place. The person the decision is applied to also bears a cost — a refused loan, an unnecessary biopsy, a missed intervention — and that cost appears nowhere in the derivation unless someone deliberately puts it there. A cost ratio is a statement about whose errors matter, presented as arithmetic.

What the impossibility result adds. One threshold derived from one cost ratio produces the disaggregated picture of Worked 2.6a: different selection rates, recalls and precisions per group, because the groups have different base rates. So the cost-optimal threshold is automatically unfair on at least two of the three criteria. You can choose per-group thresholds to close one gap, but line 3 of the proof says you cannot close them all, and per-group thresholds are themselves legally and ethically contested since they treat people differently by group attribute explicitly.

The synthesis. There is no threshold that is simultaneously cost-optimal and fair on every criterion, so the choice must be made openly rather than discovered. The defensible practice is to state the cost ratio and whose costs it represents, state which fairness criterion is being prioritised and why, report the disaggregated metrics at the chosen threshold, and record all of it in a model card. That is not a way of solving the tension. It is a way of ensuring that whoever inherits the system can see the tension and revisit the decision — which, given that the alternative is a single accuracy figure, is a substantial improvement.

Probably yes, but the accuracy figure is nearly worthless. 0.8900 against a baseline of 0.8500 is a gain of four points, not eighty-nine. The real evidence is elsewhere: balanced accuracy 0.8529 against a chance value of 0.5000, and MCC 0.6301 against 0, both say the model has substantial genuine skill that accuracy conceals.

The operational question is whether 40 false alarms is an acceptable price for finding 60 of the 75 positives. Precision 0.6000 means three in five flags are real. If a flag triggers a cheap check, deploy. If it triggers something costly or harmful, raise the threshold and recompute.

The four values run from 0.2899 to 0.6557 on the identical model, which is the point: an F-score without its β is not a measurement. Each one is pulled toward whichever component it favours, and precision here is the weak one at 0.25.

Report F₂ or F₃, and say why. This is a screening model, so a missed case is far worse than a false alarm and the metric should encode that. Better still, report precision and recall separately alongside it — 0.25 and 0.80 tell a reader everything, and a single F-score tells them nothing without the pair. Choosing β after seeing the numbers, to make the model look good, is the failure mode to avoid.

Cross-check the AUC by area: right-steps of width 1/4 occur at TPR heights 0.25, 0.75, 1.00, 1.00, so the area is (1/4)(0.25 + 0.75 + 1.00 + 1.00) = (1/4)(3) = 0.7500. ✓ The four discordant pairs all involve the negative at rank 2, which outranks three positives — a single badly-scored negative accounts for every error.

Investigate anything above a 7.7% chance of fraud. That is a long way from 0.5 and it is exactly what a 12:1 cost ratio demands — you are willing to open twelve investigations to catch one fraud, because the thirteenth would break even.

Note that the answer is 1/13, not 1/12: the formula divides by the sum of the costs. Note also that neither answer involves the prevalence, however rare fraud is, provided the model's probabilities are calibrated. If they are not, both numbers are meaningless and 2.4.6 comes first.

RMSE 1.5811 against MAE 1.5000 is a narrow gap, consistent with errors of only 1 and 2 — evenly sized, no outliers. MAPE's 15.75% is dominated by the second point, where an error of 2 on a true value of 7 is 28.6%; the same error on the value of 18 is 11%. That is MAPE's characteristic distortion and the reason it should not be the headline figure when the target spans a wide range.

The 0.9 bin is much worse, and it is overconfident: the model claimed 90% and delivered 70%. The 0.6 bin is very slightly underconfident, claiming 60% and delivering 65%, which is a gap of 0.05 and well within noise on 60 cases.

So the pattern is confidence inflated specifically at the top end, which is the signature of an overfitted or unregularized model — and the most consequential place to be wrong, since high-confidence predictions are the ones acted on automatically. A monotone recalibration would pull the 0.9 bin down toward 0.7 while barely touching the other, and it would leave the AUC exactly where it is.

Predictive parity holds exactly — both groups have PPV 0.6000, so a flag means precisely the same thing whichever group you belong to. Everything else fails, and badly. Demographic parity: 0.6000 against 0.1500, a four-fold difference. Equal opportunity: 0.9000 against 0.3000, so a qualified member of group B is three times less likely to be found than a qualified member of group A. Equalised odds fails on both components.

This is the impossibility theorem in its most uncomfortable form. A vendor could truthfully report "the model is calibrated identically for both groups, and a flag is equally reliable for everyone" while 70% of group B's qualified members are being missed. Predictive parity was satisfied because group B's threshold is effectively so high, and the algebra of PPV = p·TPR/(p·TPR + (1−p)·FPR) shows how: B's lower prevalence is offset by its far lower FPR, landing on the same PPV. Verify it: 0.3(0.3)/[0.3(0.3) + 0.7(0.0857)] = 0.09/0.15 = 0.6000. ✓

Which criterion you should have picked depends on the harm. If the flag grants a benefit, equal opportunity is the one that matters and this model is indefensible.

The honest sentence: "five-fold cross-validated accuracy 0.750, standard deviation 0.057 across folds, 95% interval 0.679 to 0.821."

To claim the rival is worse you need its per-fold scores on the identical folds. A rival mean of 0.77 is inside this model's interval, so an unpaired comparison can conclude nothing. But if the folds are shared, compute the five paired differences and test their mean against zero — fold-to-fold difficulty cancels and the test becomes far more sensitive, as Worked 2.4.6 showed with a mean difference of just 0.018 reaching t = 4.81. Note the direction here, though: 0.77 exceeds 0.75, so the rival is ahead on the means and the burden is on you.

Four times the weight gives one quarter the uncertain band, since the band width is 2 ln 9 / w. So this model pseudo-labels almost everything, while the flatter model of Worked 2.5 pseudo-labelled almost nothing.

And that is the danger, not the benefit. A large weight vector means the model has been driven to high confidence — on separable data, per 2A section 2.3.1, to arbitrarily high confidence — without necessarily being more accurate. Its confidence threshold has stopped filtering anything, so every prediction including the wrong ones is committed as a pseudo-label, and the confirmation-bias loop runs at full speed. Before trusting a confidence threshold, check the reliability diagram; the model that most wants to pseudo-label is the one least entitled to.

The model's accuracy of 0.9626 is below the do-nothing baseline of 0.9800, so any accuracy-driven selection would reject it. Balanced accuracy 0.7850 and MCC 0.4006 say it has real, moderate skill — it finds 60 of 100 frauds. This is exactly the pattern of 2.4.2's hospital depth box.

The single change: choose the threshold from the cost ratio rather than leaving it at 0.5. Everything above describes one arbitrary operating point on a curve. If a missed fraud costs 12 times an unnecessary investigation, then t* = 1/13 = 0.0769, and at that threshold recall will be far above 0.60 at the price of more of the 147 false alarms — which is the trade the costs say to make. Report the AUC and average precision for the model, then the confusion matrix at the chosen threshold, and quote the prevalence beside both.

Worth noting what the specificity of 0.97 conceals: it sounds excellent and produces 147 false alarms, because 3% of 4,900 is a large number. When negatives dominate, specificity is the wrong lens and precision is the right one.