Risk management

Monte Carlo and luck in backtests: skill or noise?

How reshuffling and resampling trades shows how much of a backtest was luck, exactly what Veloce's luck panel computes, and what Monte Carlo cannot tell you.

By
Veloce team
Published
Reading time
8 min read
On this page
  1. Why one backtest is not enough
  2. Reshuffling: same trades, different order
  3. Bootstrapping: similar trades, different total
  4. Exactly what Veloce’s luck panel computes
  5. A worked example
  6. More trades narrow the range
  7. One big trade
  8. What Monte Carlo cannot tell you
  9. Other ways to separate skill from noise
  10. Questions traders ask
  11. A checklist
  12. Try it in Veloce

A backtest gives one result from one path through history. Monte Carlo simulation asks how different that result could have been with the same kind of trades: reshuffle the trades to see how deep the drawdown could have gone, and resample them with repeats to see how widely the total could vary. If a profit survives most resamples, the trades carry some consistency; if a large share of resamples lose money, the profit may be luck. This guide explains both methods, describes exactly what Veloce’s “How much was luck” panel computes, works through an example with the panel’s own code, and sets out what Monte Carlo cannot tell you.

Why one backtest is not enough

Suppose a strategy made ₹4,930 over 20 trades, with a worst drawdown of ₹1,100. Two separate questions hide in that line:

  1. Was the drawdown typical? The same 20 trades in another order could have produced a much deeper fall before the recovery. Live trading will deliver the trades in some order, and not necessarily a kind one.
  2. Was the total typical? If the strategy’s trades are a sample from what it tends to do, another sample of 20 could have totalled more or less. With few trades, one or two big winners can carry the whole result.

Monte Carlo answers each question with a different resampling method.

Reshuffling: same trades, different order

Reshuffling (a permutation test on the sequence) keeps every trade exactly once and randomises the order. The total never changes. Only the path changes, and with it the maximum drawdown: the largest fall in cumulative P&L from a peak.

Run enough shuffles and you get a distribution of drawdowns the same trades could have produced. Compare the drawdown that actually happened with the middle and the bad end of that distribution.

Bootstrapping: similar trades, different total

Bootstrapping draws as many trades as the backtest had, at random and with replacement, from the backtest’s trade list. Some trades appear twice, some not at all. Each draw is a plausible alternative set of trades under the assumption that the past trades are a fair sample. Summing each draw and repeating gives a distribution of totals.

If most totals are positive, the profit does not rest on a few lucky trades. If a large share are negative, it might.

reshuffle: order changes bootstrap: draws with repeats ABCDE DAECB ABCDE CACEC same total, new drawdown B and D missing, C three times: new total
Reshuffling tests the path; bootstrapping tests the total.

Exactly what Veloce’s luck panel computes

Every backtest report in Veloce with at least 10 closed trades includes a panel titled “How much was luck”. Its description reads “1,000 reruns of these N trades: first in shuffled order, then drawn at random with repeats.” Here is what the code does.

The input. The list of closed trades in the report, each trade’s net P&L in rupees after its own charges, in the order the trades closed. A trade is one contract opened and closed, so each leg of a multi-leg position (the four legs of an iron condor, say) is a separate trade.

The runs. 1,000 runs. Each run does two things:

  1. Shuffles the full trade list with a Fisher–Yates shuffle and measures its maximum drawdown.
  2. Draws N trades at random with replacement, where N is the number of trades, and sums them.

The drawdown. Cumulative P&L starts at zero. The peak starts at zero and rises with the cumulative P&L; the drawdown is the largest gap between the peak so far and the current cumulative P&L. It is measured on closed-trade P&L only, so it ignores swings while positions are open and is usually smaller than the report’s equity-curve drawdown.

The random numbers. A fixed-seed generator (mulberry32), so the same trades always give the same figures. Rerunning a report does not change the panel.

The percentiles. The 1,000 values are sorted from lowest to highest and the value at position ⌊share × 1,000⌋ is taken: the 500th for the median, the 950th for the 95th percentile, the 50th for the 5th percentile (counting from zero).

What it shows.

Section Row Meaning
Drawdown in another order In the order they happened Maximum drawdown of the trades in their real order
Typical Median drawdown of the 1,000 shuffles
Worst 1 in 20 95th percentile of the shuffled drawdowns
Net P&L of similar trades Low end (1 in 20 worse) 5th percentile of the 1,000 resampled totals
Typical Median resampled total
High end (1 in 20 better) 95th percentile of the resampled totals
Chance of a net loss Share of resampled totals below zero

Drawdowns are shown as negative rupee amounts.

A worked example

Here are 20 illustrative trade results, in rupees after charges, in the order they closed:

1,450, −620, −480, 2,300, −710, 380, −540, 1,880, −650, −300, 960, −720, 2,600, −590, −410, 740, −680, 1,210, −560, −330

Eight winners and twelve losers: a 40% win rate, an average win of ₹1,440 against an average loss of ₹549, a profit factor of 1.75 and a total of ₹4,930. A trend-following option buyer often looks like this.

Running these through the panel’s code gives:

Row 20 trades
Drawdown in the order they happened −₹1,100
Typical drawdown −₹2,610
Worst 1 in 20 drawdown −₹4,150
Low end total (1 in 20 worse) −₹2,650
Typical total ₹4,730
High end total (1 in 20 better) ₹12,640
Chance of a net loss 15%

Read it line by line:

  • The real drawdown was kind. ₹1,100 is less than half the typical shuffled drawdown and about a quarter of the 1-in-20 figure. The winners happened to be spread between the losers. Plan for a drawdown nearer ₹4,150 than ₹1,100.
  • The total is not secure. One resample in seven lost money, and the low end is a loss of ₹2,650. Twenty trades at this win rate cannot distinguish a modest edge from luck.

More trades narrow the range

Now take the same 20 trades repeated three times, as if the strategy had traded for three times as long with the same mix of results:

Row 20 trades 60 trades
Total ₹4,930 ₹14,790
Typical drawdown −₹2,610 −₹4,140
Worst 1 in 20 drawdown −₹4,150 −₹6,730
Low end total −₹2,650 ₹1,520
Typical total ₹4,730 ₹15,140
High end total ₹12,640 ₹29,660
Chance of a net loss 15% 3.2%

The average trade is the same, ₹246.50, but with 60 trades the chance of a net loss falls from 15% to about 3%, and even the low end is a profit. The drawdown grows, because a longer run gives more room for losing streaks. This is the trade-off a longer backtest shows: more confidence in the total, a deeper worst case to sit through.

One big trade

Replace the ₹2,600 winner with a single ₹9,000 winner and leave the rest:

Row Original With one ₹9,000 trade
Total ₹4,930 ₹11,330
Typical total ₹4,730 ₹9,550
Low end total −₹2,650 −₹2,030
Chance of a net loss 15% 11%

The total more than doubled, but the chance of a loss barely moved. Without that one trade, the remaining 19 total ₹2,330. A draw of 20 trades with repeats leaves the big trade out about a third of the time ((19 ÷ 20)²⁰ ≈ 36%), which is why the low end stays negative. When a single trade carries a result, the panel’s low end and loss chance show it even when the total looks strong.

What Monte Carlo cannot tell you

The panel answers “how much could this vary if my trades are a fair sample?” It cannot answer whether they are.

  • It assumes independent trades. Real trades cluster: losing streaks come in range-bound months, winning ones in trends. Shuffling breaks those clusters, so the shuffled drawdowns can understate the risk when losses really do bunch together.
  • It treats legs as separate trades. An iron condor’s four legs close together and their results are linked. Shuffling them independently spreads correlated losses apart. For multi-leg strategies, read the panel’s drawdown as optimistic and look at drawdown by day or by expiry in the report.
  • It uses only the market you tested. Every resample comes from the same months. If those months were unusually calm, every resample is calm too.
  • It cannot see the selection. If the strategy was the best of 16 combinations in a sweep, its trades were chosen for being good. Monte Carlo on the winner’s trades does not undo that; a check period does. See the walk-forward guide.
  • It does not change costs. Charges and slippage are inside each trade’s P&L. If real slippage is higher, every trade is worse, which no resample reveals. The slippage guide covers testing that.

Other ways to separate skill from noise

Compare with a no-signal baseline. A strategy spec with no signal enters its legs whenever the rest of its entry rules allow. Run the same legs, entry times, exits and square off with and without the signal. If the signal does not beat entering anyway, it adds nothing.

Check a later period. A held-back check period tests the settings on days they were not chosen on.

Look for a plateau. If nearby settings give similar results, the result depends less on luck in the exact numbers.

A rough t-statistic. The average trade divided by its standard error. For the 20 example trades, ₹246.50 ÷ ₹248.10 ≈ 1.0; for the 60, about 1.75. Values near 2 or more are needed before a trader would usually call an average reliably above zero, and even that assumes independent trades.

Questions traders ask

Should I size positions from the worst 1-in-20 drawdown? It is a better planning figure than the historical drawdown, but not a limit. Losses can exceed it, and for multi-leg strategies it is optimistic. The drawdown and sizing guide covers sizing.

Why does the typical total differ from the actual total? The median of 1,000 bootstrap totals is close to the actual total but not equal to it, because resampled totals are spread around it and the distribution is not perfectly symmetric.

Is 1,000 runs enough? For medians and 1-in-20 percentiles, yes: rerunning with a different seed moves them only slightly. Much rarer events (1 in 1,000) would need more runs and, more importantly, more trades.

A checklist

Check Why
Plan for the shuffled 1-in-20 drawdown, not the historical one History’s order was one of many
Read the chance of a net loss It shows whether the total rests on a few trades
Remove the largest trade and recount One outlier can carry a result
Treat leg-level shuffles of multi-leg strategies as optimistic Linked legs lose together
Compare with a no-signal baseline Shows what the signal adds
Confirm on a later period Monte Carlo cannot undo selection

Try it in Veloce

Every Veloce backtest on NIFTY and BANK NIFTY options, SENSEX options or MCX gold futures with ten or more closed trades shows the “How much was luck” panel beside the equity curve, drawdown and itemised charges. Reports explains each figure, the backtesting engine page explains how the trades were filled, and your run history keeps every result for comparison.

Create a free Veloce account and read the luck panel on your next backtest before you read the total. For a full testing method, see how to backtest an options strategy.

Frequently asked questions

What does a Monte Carlo simulation of a backtest do?
It reruns the backtest's own trade results many times in a different way: either in a shuffled order, to see how deep the drawdown could have been with the same trades, or drawn at random with repeats, to see how much the total could vary with similar trades. It uses no new market data.
What is the difference between reshuffling and bootstrapping trades?
Reshuffling keeps every trade exactly once and changes only the order, so the total P&L is unchanged and only the path, and hence the drawdown, varies. Bootstrapping draws trades at random with replacement, so some trades appear twice and others not at all, which makes the total vary.
How many trades do I need for a Monte Carlo test to mean anything?
There is no fixed minimum, but with a few dozen trades the ranges are wide and one large trade can dominate. Veloce shows its luck panel only from 10 closed trades, and the figures become more stable as the count grows.
What does 'chance of a net loss' mean in Veloce?
It is the share of 1,000 resampled sets of trades, each the same size as your backtest's trade list and drawn with repeats, whose total net P&L was below zero. It estimates how often trades like yours would have added up to a loss by chance.
Can Monte Carlo prove a strategy has an edge?
No. It assumes your trades are a fair sample of what the strategy will do and that each trade is independent of the others. It says how much the result could vary if that is true; it cannot say whether it is true.
Why is the Monte Carlo drawdown bigger than my backtest's drawdown?
Because the order your trades happened in was one of many possible orders, and often a kind one. Shuffling puts losing trades next to each other more often than history did, which deepens the typical drawdown.

Sources

  1. AmiBroker User's Guide: Monte Carlo simulation of your trading systemamibroker.com, accessed
  2. MQL5: Stress testing trade sequences with Monte Carlo in MQL5mql5.com, accessed
  3. Wikipedia: Bootstrapping (statistics)en.wikipedia.org, accessed
esc
↑↓Move↵Open