Calibration and Brier Score: How to Tell a Real Probability From a Number

Anyone can print a percentage. The question that separates a forecast from a decoration is whether that percentage means anything — and there is a precise, testable answer.

Calibration in one sentence

A forecast is calibrated when its stated probabilities occur at the stated rate.

Collect every match the model marked 60%. If roughly 60% of them happened, the forecast is calibrated at that level. Collect everything marked 30%; roughly 30% should have happened.

Two properties follow immediately:

• You cannot test calibration on one match. A 60% forecast that lost tells you nothing at all.
• You can test it thoroughly on hundreds. And the test is not subjective.

Why hit rate is a weak measure

Hit rate is the number products quote, and it is close to meaningless on its own.

Consider two models over a hundred matches:

	Model A	Model B
What it forecasts	Only heavy favourites, always around 80%	Full range, 25%–75%
Hit rate	74%	53%
Calibrated?	Unknown from this number	Unknown from this number

Model A looks dramatically better and has told you almost nothing you did not already know from the odds board. Model B covers matches where the answer is genuinely uncertain, which is where a forecast is worth having.

Hit rate rewards forecasting the obvious. Calibration rewards being honest about how sure you are.

Brier score

Brier score is the mean squared error of a probabilistic forecast. For each event, take the stated probability, subtract the outcome (1 if it happened, 0 if not), square the difference, and average.

Brier = mean( (probability − outcome)² )

Lower is better. Some reference points:

Forecast	Outcome	Squared error
0.90	happened	0.01
0.90	did not	0.81
0.60	happened	0.16
0.60	did not	0.36
0.50	either	0.25

Look at the second row. Stating 90% on something that does not happen costs 0.81 — more than three times the cost of simply saying 50% and being wrong. Squaring the error is what makes confident wrong answers expensive, and that is exactly the behaviour a scoring rule should discourage.

A useful anchor: a forecast that always says 50% on a two-way market scores 0.25. Anything above that is worse than admitting you do not know.

Log-loss, and why it drives model weighting

Log-loss takes the punishment further:

log-loss = −mean( log(probability assigned to what happened) )

As the stated probability for something that then occurs approaches zero, log-loss approaches infinity. It is brutal about confident errors in a way Brier is not.

That severity is the reason ensemble weights are grid-searched to minimise log-loss rather than to maximise hit rate. Optimising for hit rate would push the ensemble toward safe favourites; optimising for log-loss pushes it toward honest uncertainty.

Two rules keep this from becoming self-flattery:

• The weights for any test season are fitted only on prior seasons.
• The boosting component's contribution to training predictions comes from a nested split, so it does not grade its own homework.

How calibration is actually fixed

Raw output from an ensemble of models is almost never calibrated. The usual failure is overconfidence at the extremes: the model says 80% and the observed rate is 72%.

The correction is isotonic regression, applied per outcome. It learns a monotonic mapping from stated probability to observed frequency without assuming any particular shape — which matters, because the distortion is rarely a neat curve.

One constraint: it runs only where there are at least 300 samples. Fitting a calibration map on fifty observations produces a map that describes those fifty observations and nothing else.

What the published backtest shows

The backtest runs across five major European leagues, walking forward season by season. Two results, published as they stand:

1. The ensemble reaches roughly 52–53% accuracy on 1X2 across test seasons.
2. The market's closing price, after margin removal, still measures about 0.02–0.03 better in log-loss than the model-only ensemble.

The second point is the one that matters, and it is the one most products would omit. It says the market is an excellent probability estimator, and that the boosting component — which earns meaningful weight in most seasons — does not close the gap.

The named next levers are equally specific: fold xG into the feature set, add player availability and lineup adjustments, and tune the update rate and time decay per league rather than globally.

Why this drives the whole product design

If the market is better calibrated than the model on average, what is the model for?

Not for beating the market everywhere. For finding the specific fixtures where the two disagree by more than the estimate's own error bar — and for saying nothing everywhere else.

That is why the required gap grows with every measurable source of uncertainty, why six of the thirteen decision states are explicit PASS states, and why a typical round produces very few actionable fixtures. The selectivity is not caution for its own sake. It is the direct consequence of an honestly measured deficit.

How to test any forecaster yourself

Five steps, no tools beyond a spreadsheet:

1. Record every published forecast with its stated probability, before kickoff.
2. Bucket them: 30–40%, 40–50%, 50–60%, and so on.
3. For each bucket, compute the observed rate.
4. Compare the observed rate to the bucket midpoint.
5. Compute Brier score across the whole set and compare it to 0.25.

If the buckets track their midpoints and Brier sits meaningfully below 0.25, you are looking at a forecast. If only a hit rate is available and the buckets are unpublishable, you are looking at marketing.

The reason to prefer calibration over everything else

A calibrated probability is the only kind you can put against a price. Fair odds are one divided by the probability; if the probability is not calibrated, the fair odds are fiction, the edge is fiction, and every downstream decision inherits the error.

Everything else in a decision — the uncertainty budget, the minimum acceptable price, the decision state — assumes that the number entering it means what it says. Calibration is the assumption that makes the rest of the machinery legitimate.

A five-step test you can run on anyone

1. Record every published forecast with its stated probability, before kickoff.
2. Bucket them: 30–40%, 40–50%, 50–60%, and so on.
3. Compute the observed rate in each bucket.
4. Compare each observed rate to its bucket midpoint.
5. Compute Brier score across the whole set and compare it to 0.25.

If the buckets track their midpoints and Brier sits meaningfully below 0.25, that is a forecast. If only a headline hit rate exists and the buckets cannot be produced, that is marketing.

---

18+. WinPIQ is an analysis tool, not advice and not a promise. Betting can be addictive and money can be lost. Only stake what you can afford to lose, and if betting stops being entertainment, seek help. WinPIQ is not affiliated with Winner or the Israeli Council for the Regulation of Sports Betting.