In one sentence
Dynamic programming breaks a decision that unfolds over several steps into small pieces and solves them backwards from the last step, so each choice accounts for everything that can happen after it.
How it works
Many betting decisions are about timing. Should you take 5.0 now, or wait in case 5.4 shows up? Should you trade out now or hold on? The right answer depends on what you could do later, and that depends on what you could do after that.
Dynamic programming, developed by Richard Bellman in the 1950s, cuts through this by starting at the end. At the final moment there is no choice left, so the value is easy to compute. One step earlier, you compare "act now" with the value of waiting, which you have just worked out. Repeat back to the start.
The same method is behind working out in-play win chances from any score and minute, optimal stopping problems and much of reinforcement learning.
The maths
- Vt(s): the best expected value you can achieve from state s at time t.
- a: an action you can take, such as "bet now" or "wait".
- r(s, a): the immediate reward from taking action a in state s.
- P(s′ given s, a): the chance of moving to state s′ next.
- Vt+1(s′): the best value from the next state, already worked out.
In plain English: the value of being here is the best of your options, each scored as its immediate payoff plus the expected value of where it leads.
Worked betting example
Your model rates the away win in a Match Odds market at 22.2% (fair odds 4.5). You must get £100 on before kick-off, and you will check the price three times. Each time the best back price is 4.6, 5.0 or 5.4 with illustrative chances of 30%, 40% and 30%. (A simplification: real prices are linked from one check to the next.)
Expected value per £1 at each price: (1 ÷ 4.5) × odds − 1.
- 4.6: +0.022
- 5.0: +0.111
- 5.4: +0.200
Check 3 (last chance, must bet): value = 0.3 × 0.200 + 0.4 × 0.111 + 0.3 × 0.022 = 0.111.
Check 2: take the price only if it beats 0.111 from waiting. Take 5.4; 5.0 ties so taking it is fine; wait on 4.6. Value = 0.3 × 0.200 + 0.7 × 0.111 ≈ 0.138.
Check 1: take only if it beats 0.138. Only 5.4 qualifies. Value = 0.3 × 0.200 + 0.7 × 0.138 ≈ 0.156.
So the best plan is worth about £15.64 per £100 before commission, against £11.11 for simply taking whatever shows at the first check. With 2% commission on winnings the same working gives about £13.78 against £9.33. The policy: at the first check be fussy, at the second accept 5.0 or better, at the last take anything.
Where it's good
- Timing entries and exits when you have several chances to act.
- In-play football pricing, working back from every possible final score; the same trick prices tennis from any score.
- Managing a trading position through a match with rules for when to green up.
- Staking over a series of bets with a target or a limited bank.
Limitations and pitfalls
- You need a model of how states change, and in betting that model (how prices move) is usually the weakest link.
- The number of states can explode: adding price, time, position and liquidity quickly makes the problem too big.
- Assuming independent prices, as above, ignores that a drift often signals bad team news.
- The answer is optimal only for your model; a wrong model gives a confidently wrong policy.
- Commission and the chance of missing kick-off entirely need to be built in, or the plan is too optimistic.
How to build it
- Small problems need only a few lines of Python with numpy arrays or a cached recursive function.
- For larger problems, approximate methods from reinforcement learning libraries (such as stable-baselines3) take over.
- Tip: sanity-check the policy on simulated prices before trusting it with real money.
Related methods
- Markov chains supply the state transitions dynamic programming works over.
- Reinforcement learning extends the idea when transition chances are unknown.
- Expected value is the reward being maximised at each step.
- Queue position is another timing choice between waiting and acting now.