How to Evaluate an AI Football Prediction App: Seven Questions

The "AI" label has become meaningless in this category. It can sit on a model built over years, and it can sit on an interface that displays the market price in a different colour.

Here are seven questions that separate them. None require technical knowledge.

1. Are market prices an input to the model?

Why it is critical: if the price is a feature, the model learns to imitate the market. It can still be useful as a data display, but any "gap" it reports is noise around the consensus, because it was built from the consensus.

How to check without asking: look at ten fixtures. If the product's probabilities are consistently very close to the price-implied probabilities, and the recommendation always agrees with the favourite, that is likely what is happening.

Our position: market prices are never a feature, at any stage. A context-adjusted estimate blending the model with a no-vig market prior does exist internally, but it is never shown as the prediction — because in that blend every edge is erased by construction.

2. How were the parameters chosen?

Why it is critical: if weights or rules were selected on the same data the accuracy is reported from, the number is systematically inflated. This is the single most common mistake in amateur sports modelling.

What it should be: strict separation. Weights for a given test season fitted only on prior seasons, with the backtest walking forward season by season.

Warning sign: a backtest reporting unusually high accuracy — for instance well above 60% on 1X2 in major leagues. Published walk-forward numbers in this space tend to sit in the low fifties. Anything dramatically higher is almost always leakage rather than genius.

3. How is leakage prevented?

What it is: training or computing that unintentionally uses information that did not exist at prediction time. The result is an impressive backtest and a worthless product.

What it should be: a mechanism that fails explicitly when a feature is dated on or after the as-of moment, not merely good intentions. In our pipeline, feature construction raises an error in exactly that case, and the snapshot builder filters and asserts.

How to ask: "How do you ensure features contain no post-match information?" A vague answer is an answer.

4. Are the probabilities calibrated?

Why it is critical: a high hit rate is trivially achievable by only predicting heavy favourites. The real question is whether 60% happens about 60% of the time.

What it should be: a public breakdown of published probabilities against actual outcomes, in buckets — plus a proper scoring rule such as Brier score, which punishes confident wrong answers.

Warning sign: a product showing only an overall hit rate. That number can be manufactured several ways and cannot be interpreted.

5. What happens when there is no data?

Why it is critical: a product that emits a number for every fixture on earth, including leagues with no history and no price depth, is manufacturing confidence.

What it should be: a visible data quality score and an explicit flag below a threshold. In our case the score is built from history 35, odds depth 15, xG 15, lineups 15, player data 10, injuries 5 and news 5. A fixture with results only and two price sources scores about 41, and below 40 it is flagged as data-limited. Goals markets are published only when both teams have at least 20 matches of history.

6. Where is the record, including the losses?

Why it is critical: without a complete record, any performance claim is uncheckable. And football outcomes are noisy enough that a good run happens to everyone.

What it should be: a public record of every settled prediction, wins and losses in the same table, with timestamps, the price at lock, and a way to verify nothing was altered. In our case that is a SHA-256 hash over a canonical payload, stored in a ledger where any update or delete attempt fails at the database level.

Warning sign: screenshots, a channel that can be cleaned, or "results" with no total prediction count.

7. Is the product willing to say "nothing today"?

Why it is critical: a product that must produce a selection for every fixture every day is producing content, not analysis. A typical round contains very few situations where estimate and price genuinely separate.

What it should be: a reasoned abstention state that is published and counted. We use thirteen decision states, six of them PASS states with distinct reasons, plus a weekly selectivity report computed from real rows.

Summary table

Question	Good answer	Red flag
Market prices as input?	No	Probabilities always hug the price
Parameter selection	From prior data only	Unusually high backtest accuracy
Leakage prevention	A mechanism that fails explicitly	"We are careful"
Calibration	Public buckets plus Brier	Overall hit rate only
Missing data	Visible score plus a flag	A number for every fixture on earth
Record	Public, hashed, includes losses	Screenshots
Abstention	Reasoned PASS, counted	A selection for every fixture, daily

Three patterns that recur in weak products

The probabilities look too smooth. A product showing 55/25/20, 60/22/18, 50/27/23 for every fixture — always round, always plausible, never surprising — is probably generating them from a simple rule rather than a model. A real model produces awkward numbers, and occasionally produces numbers that look strange.

There is no middle state. Every fixture gets a recommendation, and recommendations differ only by star rating. No "insufficient data", no "wait for lineups", no "price too short". A system without intermediate states was not built to abstain.

The explanation always fits the result. After the match, the explanation shifts. Before it, the explanation is general enough to fit any outcome. The test: read the pre-match explanation and ask whether it would have sounded equally reasonable for the opposite result.

A two-week test that costs nothing

If you are considering any product:

1. Pick twenty fixtures in advance, across different leagues.
2. Before each, record the product's probability, the market price and the recommendation.
3. Act on nothing. Just record.
4. After two weeks, check three things: how often the recommendation differed from the market favourite, how often the product said there was nothing to do, and whether the probabilities distribute sensibly against outcomes.

That test is worth more than any review, and it costs nothing.

The hardest thing to fake

Finally, the test no marketing department passes voluntarily: a product that publishes a finding that does not flatter it.

For us that is the finding that the market's closing price, after margin removal, still measures roughly 0.02–0.03 better in log-loss than our ensemble in the walk-forward backtest. It is displayed as it stands, because concealing it would contradict everything above.

---

18+. WinPIQ is an analysis tool, not advice and not a promise. Betting can be addictive and money can be lost. Only stake what you can afford to lose, and if betting stops being entertainment, seek help. WinPIQ is not affiliated with Winner or the Israeli Council for the Regulation of Sports Betting.