The best of 480 backtests

The short answer

Pick the best of many backtests and it will look better than it is. GEXfreak had an AI model draft trading rules and back-tested 480 versions of them on 12 stocks, 2021–2024, on recorded prices; five passed all seven tests of our pass mark. Finding those took 864 side-by-side tests (three for each of the 288 rules we asked the model for), so luck alone was expected to produce a result this good: the best p-value was 0.00215 (about 1 in 465), and among the 665 usable results you expect 665 × 0.00215 = 1.4 that small (1.0 counting only the 480 tests against cash). Adjusted for the tries, none of the five could be told apart from luck.

The routine that follows works on any backtest, yours or someone else’s: count the tries, count the trades, turn the result into a t-score, compare it with the bar for that many tries, then test on data the search never saw. The study’s own limits come last: it was not a fresh test, its stocks were hand-picked, and every window on this page ends on 31 December 2024, with no data after that date used on the page.

Score any backtest, including your own

Five steps, in order. Use each trade’s % return or R, not dollars, so one large trade does not drown the rest. Most platforms export a trade list; % return = profit ÷ (entry price × shares), or profit ÷ premium paid for an option bought outright, the most you can lose for a spread, or the margin posted for a future.

• 1. Count the tries. Every stock, setting or time frame you changed and re-ran counts as one, including the ones you threw away: 20 settings on 3 stocks is N = 60. If you cannot recount, use the next row up in the table below. This round’s 480 versions needed a bar of 3.88, or 4.02 if you count all 864 comparisons it set up.
• 2. Count the trades behind the winner. Under 30 there is nothing to judge; the exact bar for a few trades is higher (see the table). The five that passed had 32 to 58.
• 3. Turn the result into a t-score: your average trade divided by how uncertain that average is (the standard deviation of your trades ÷ √trades). About 2 is the usual line for a single test, the same as the 95% range just clearing zero; more tries raise the line. With each trade’s return in B2:B41: =AVERAGE(B2:B41)/(STDEV(B2:B41)/SQRT(COUNT(B2:B41))). If your tool prints a 95% range instead: the average ÷ ((high − low) ÷ 3.92). If it prints a p-value: =NORM.S.INV(1-p/2). If it prints only an annual Sharpe ratio: the Sharpe ratio × √(years of data). To turn a t-score back into a p-value: =T.DIST.2T(t,COUNT(B2:B41)-1).
• 4. Compare it with the bar for your tries: =T.INV.2T(0.05/N,COUNT(B2:B41)-1), where N is your count from step 1 (with many trades it matches =NORM.S.INV(1-0.025/N)). The table has it for common counts. A t-score under the bar is a lead, not a finding. The average you would need to clear it is =bar*STDEV(B2:B41)/SQRT(COUNT(B2:B41)).
• 5. Test it on data the search never saw. Hold back the latest year before you tweak anything, and test your final rule on it once; this round did not. If you have already seen all your history, log new trades on paper instead: fix the number first (“How many trades until you know?” shows how; 30 trades is about seven months at one a week), check once at the end, and use =T.INV.2T(0.05,COUNT(B2:B41)-1) as the bar (2.05 at 30 trades), because a fresh test of one frozen rule is a single try. Until then it is a lead, not a result.

A worked example

This round’s best-looking version, against holding cash: 58 trades that averaged +0.812% with a standard deviation of 2.409%. Its t-score is =0.812/(2.409/SQRT(58)) = 2.57, and =T.DIST.2T(2.57,57) gives p = 0.013. Its bar at 864 tries and 58 trades is =T.INV.2T(0.05/864,57) = 4.35, so it falls well short. The average it would have needed is 4.35 × 2.409 ÷ √58 = 1.38% a trade, against +0.812%.

The study’s own p-value for this version, 0.00215, comes from re-drawing its daily returns rather than scoring its trades, and converts to =NORM.S.INV(1-0.00215/2) = 3.07. Two ways of scoring the same version disagree, and the p-values below look too small, so use the trade-based t-score for your own backtest.

How the 480 were made

The model was called 72 times, once for each stock and idea family, and each call asked for 4 rules: 12 × 6 × 4 = 288 slots (a slot is one requested rule). 276 came back and 240 were testable (12 slots came back empty, 36 were rejected before testing). Each ran twice, plain (its core setup alone) and filtered (plus the one extra filter the model proposed), so 480 versions, each on one stock and held at most 5 sessions (“weekly”) or 21 sessions (“monthly”), leaving earlier if a stop or target was hit. Commission $0.005 per share per side; slippage of 0.02% of price on the largest stocks or 0.05% on the others, never less than one cent a share; dividends included. Every version was rerun at double those costs. Its limits, from the study’s protocol, fixed before the run: The twelve stocks are a fixed, hand-picked set, not the whole market. Gamma is modeled from option data, not observed dealer positions. The monthly volume profile is an approximation from one-minute bars. The interest-rate data is lagged and uses the latest revised values, not the numbers as first published. The six families of idea:

• Value-area reversion. Trading back toward, or away from, the prior month's most-traded price range.
• Trend continuation. Buying pullbacks and recoveries inside an uptrend.
• Breakout acceptance. Buying a fresh breakout that holds.
• Failed breakdown. Buying a recovery after a drop failed to hold.
• Gamma structure. The same price setups filtered by modeled gamma sign or by how far the call and put walls sit.
• Macro resilience. The same price setups filtered by interest-rate and market-drawdown context.

The pass mark

A version had to pass all seven tests. The 95% range is the band the true average trade probably sits in: if its low end is below zero, luck could explain the average.

• 1. At least 30 trades, spread over at least three years.
• 2. A positive average trade whose 95% range stays above zero.
• 3. A profit factor of at least 1.3, and at least 1.1 when costs are doubled.
• 4. At least three profitable calendar years.
• 5. A worst dip of 15% or less.
• 6. A better average than holding SPY over the same days.
• 7. No single winner more than a quarter of all the gains.

What the tests left standing

Of the 480 versions, 68 had 30 trades or more (412 had too few, 32 of them none), 44 had a profit factor above 1 (winning returns ÷ losing returns), 37 reached 1.3, 5 passed every test and 0 passed after adjusting. Of the 37 at 1.3 or more, 32 failed another test (27 on “95% range reached zero”, 26 on “worst dip over 15%”, 6 on “fewer than three profitable years”, 4 on “no better than SPY”; some failed more than one). The five had 32 to 58 trades and profit factors of 1.85 to 2.73: on a leaderboard they would look like strategies.

Counting the tries

864 is not 480. Each of the 288 slots set up three comparisons: plain against cash, filtered against cash, filtered against plain. 144 belong to the 48 slots that never produced a testable rule, 55 more had nothing to test (32 versions never traded, so had no return to compare with cash, and in 23 filtered-versus-plain pairs the trades were identical (20) or absent (3)), and 665 gave a usable result. We adjust for all 864 to be strict.

Whichever count you prefer, the best p-value is about what luck gives. Among the 480 versions tested against cash, 0.00215 × 480 = 1.03 results that small are expected; 2 occurred (the winner and a version that lost to cash).

What adjusting does

A p-value is the chance of an average this far from zero, in either direction, if the true average were zero (holding cash), counting days out of the market as zero. Here it comes from re-drawing 63-session stretches of each version’s daily returns 20,000 times. The smallest p in the whole round, 0.0002, belonged to a version that lost: 54 trades, profit factor 0.295, −0.115% a day against cash. Two-sided p-values reward a convincing loser as much as a convincing winner.

The study used two yardsticks: the pass mark grades each trade, and the p-value grades daily returns against cash, so a version can clear one and miss the other. Adjust only for the five you are shown and two look real (adjusted p 0.011 and 0.046); adjust for all 864 comparisons and none do: every one is 1.00. An adjusted p-value stops at 1.00, so 1.00 means the correction wiped the result out. Two of the five never reached 5% even before adjusting (p 0.091 and 0.153). The bar in step 4 is the simple one (Bonferroni); for the single best result, Holm’s method, which the study used, sets the same bar.

The p-values themselves look too small, not too large. Of the 480 versions, 18 beat holding cash at p below 0.05 and 23 lost to it, where well-calibrated p-values and no real effect would give about 12 on each side. So the adjustment is strict, but the numbers feeding it look better than they should for 0.00215. Read the result as unproven, not disproven.

Sources: S. Holm, Scandinavian Journal of Statistics 6(2): 65–70 (JSTOR): A simple sequentially rejective multiple test procedure; Harvey, Liu and Zhu (NBER working paper 20592, 2014; Review of Financial Studies, 2016): “… and the Cross-Section of Expected Returns”; Bailey, Borwein, López de Prado and Zhu (working paper, 2015): The Probability of Backtest Overfitting.

Did the gamma and macro filters help?

Each filtered version can be set against its plain version. Where both traded (211 comparisons), the filter beat the plain form at p below 0.05 in 11 and lost to it in 2; luck alone would give about 5 each way. That is a hint, not a finding: these p-values are not adjusted. For gamma structure, the family that adds a modeled gamma filter, it helped in 2 of 27 and hurt in 1 (13 more had a form that never traded); for macro resilience, in 3 of 41 and hurt in none.

The five that passed came from trend continuation (2), breakout acceptance (1), gamma structure (2). Both gamma-structure passers were the plain price setups, so the gamma filter played no part in them. One of the five was a filtered version, the second form of a rule whose plain form also passed.

What was not fresh

This was not a fresh test. Earlier GEXfreak research had already examined 2021-2024. The prompts used to draft the rules carried one line from that work: an earlier monthly rule had made 12 to 29 trades per stock in 2021-2024, and its calibration had failed. The model was told not to use what it remembers of those years, but it may remember them. The correction counts only this round, so it does not account for earlier searches on the same years. Nothing after 2024 was used.

Nothing from this round is published. No version is a qualified setup: the five that passed are four distinct rules (one rule passed in both its plain and filtered form), kept as written and not tuned further, as research leads, and their stock and rule names are left out on purpose, because a best-of-480 winner is not a recommendation. A fresh test would need data this search never saw. Two more limits: a trade whose full holding period would cross a calendar year-end is not taken, and one automated check for peeking at future prices did not finish and is recorded as incomplete; direct checks and a reconciliation of every completed trade path passed. The stocks were chosen by GEXfreak, all listed when the study was built, which favors long-only results.