The calibration step we kept: when making the model more accurate would have made it worse

This is a story about almost shipping an improvement.

In July 2026 we ran a maximum-breadth research sprint against our production model: 71 experiment configurations under a pre-registered harness — frozen walk-forward splits, a sealed holdout our search never touched, log-loss and Brier as the primary metrics. One finding stood out for being both clear and embarrassing: a calibration layer in our own model makes its probabilities less accurate. Removing the per-surface isotonic step improved log-loss by about 0.013 — the single largest defect the sprint found anywhere in the system. The raw model was better calibrated than the "calibrated" one.

The obvious move is to delete the layer and ship "v6.2". We didn't — and the reason is the most useful thing this project has produced in a while.

The gate

Our pick filter — recreational-book prices that disagree with our fair line by 5–12% in the 2.0–3.0 odds range — was validated on the model with the calibration layer. Probabilities feed the filter; change the probabilities and you have silently changed the filter. So before anything moved, we re-ran the filter validation on the same frozen harness windows, once with the production model and once with the candidate:

	picks	ROI at Pinnacle close (95% CI)	hit rate
Production (with the layer)	349	+14.1% (+2.7% to +25.5%)	52.4%
Candidate (layer removed)	298	−0.3% (−12.6% to +12.1%)	46.0%

The candidate's edge isn't smaller — it's gone. And the two models only share 107 picks: removing the layer doesn't clean up the same selections, it produces substantially different ones. We also checked every neighboring filter band rather than just the one we like: the production model's edge is stable across adjacent edge bands and disappears outside the odds range; the candidate's grid is flat-to-negative everywhere.

What's actually going on

The calibration layer distorts probabilities — that's why it loses on log-loss. But the pick filter only fires when our number is at least five points away from the market's. A model that hugs the market never bets, and in the same sprint the accuracy-optimal models did exactly that: zero filtered picks. The distortions the layer introduces in the mid-range are where the profitable disagreements come from. The part of our model that makes it less accurate is the part that makes it profitable.

If that sounds suspicious to you, good — it did to us too. It's a textbook Goodhart setup, and the honest read cuts both ways: the accuracy metric isn't the product, and the profitable distortion isn't obviously skill. Our own most recent validation window ran negative for the production model (−8.2% on 57 picks). The forward ledger, which is public and currently young, is what settles that question — not this analysis.

The principle we're writing down

The system is validated as a unit. Model, calibration, filter, prices — the validated object is the whole chain, not any component. A change that improves one component in isolation, however well-measured, doesn't ship unless the unit's validated properties survive re-validation. That rule is now in our decision log, and this post is the receipt for the first time it fired.

So: the model stays exactly as it is. Both experiments — the sprint that found the defect and the gate that vetoed the fix — are logged end-to-end in the research repository, and the summary lives on the receipts page alongside the calibration record it's about.