Ranked Probability Score explained for betting

In one sentence

The ranked probability score (RPS) measures forecast accuracy for outcomes that have a natural order, rewarding forecasts that put probability near the actual result, not just on it.

How it works

Football match odds have three outcomes in a natural order: home win, draw, away win. A draw sits between the other two. If you favoured the home side and the match was drawn, you were less wrong than if the away side had won.

The Brier score ignores this ordering, but RPS respects it. It builds running totals: probability of "home", then "home or draw", then "home, draw or away" (which is always 1). It compares those running totals with the same totals for the actual result and averages the squared gaps.

The RPS has been widely used to evaluate football models, including in the academic literature on predicting match results. Lower is better and 0 is perfect.

The maths

\text{RPS} = \frac{1}{r - 1}\sum_{i=1}^{r-1}\left( \sum_{j=1}^{i} p_j - \sum_{j=1}^{i} o_j \right)^2

RPS: the score for one match, from 0 (perfect) to 1 (worst).
r: the number of ordered outcomes, 3 for home, draw, away.
p: the forecast probability of outcome j.
o: 1 for the outcome that happened, 0 for the others.

In plain English: compare the running total of your forecast with the running total of the result at each step, square the gaps, and average them.

Worked betting example

Two models forecast the same match. Model A says home 50%, draw 30%, away 20%. Model B says home 50%, draw 20%, away 30%.

Suppose the away side wins, so the result is 0, 0, 1.

Model A:

Running forecast: 0.50, then 0.80. Running result: 0, then 0.
Squared gaps: 0.50² = 0.25 and 0.80² = 0.64.
RPS = (0.25 + 0.64) ÷ 2 = 0.445.

Model B:

Running forecast: 0.50, then 0.70.
Squared gaps: 0.25 and 0.49.
RPS = 0.74 ÷ 2 = 0.37.

Model B scores better because it put more weight near the actual result. Now suppose the match had been a draw instead:

Model A: running forecast 0.50, 0.80 against result 0, 1. Gaps squared 0.25 and 0.04, RPS = 0.145.
Model B: running forecast 0.50, 0.70 against 0, 1. Gaps squared 0.25 and 0.09, RPS = 0.17.

Model A wins that one. Across a full season, these per-match scores are averaged and compared with the RPS of margin-free Betfair match odds prices.

Where it's good

Evaluating 1X2 football models such as Poisson or Dixon-Coles.
Any market with ordered outcomes: total goals bands, winning margin bands, first-half goals bands.
Comparing your model with closing market prices on thousands of matches.
Ordinal models where "nearly right" should count for something, such as handicap bands.

Limitations and pitfalls

Only makes sense when outcomes have a real order. For Correct Score or first goalscorer markets, use log loss instead.
It has been criticised for how it treats the draw in football, since a draw is not simply halfway between home and away in every respect. Report log loss alongside it.
Differences between decent football models are tiny, often in the third or fourth decimal place, so you need a full season or more and a significance test.
Scoring well does not mean betting well. The market is itself a very strong forecaster, and beating its RPS slightly may not overcome commission.
Scores vary by league and season because some leagues are more predictable; compare models only on the same matches.

How to build it

Python: a few lines of numpy using np.cumsum; the penaltyblog package also includes an RPS function.
Data: your 1X2 probabilities, results, and margin-free market probabilities for the same fixtures.
Tip: check the order of your columns. Home, draw, away must be in that order, or the score is meaningless.

Brier score - the unordered version of the same idea.
Log loss - a stricter alternative that ignores ordering.
Dixon-Coles - a classic football model usually judged with RPS.
Multinomial and ordinal logistic regression - models that output ordered probabilities.
Calibration - a separate check that the probabilities themselves are unbiased.