Judge whether a backtest result is likely to hold up out of sample.
Marcus sees a video in their feed about a moving average rule that beat the market by 6% a year for twenty years (an example figure). The presenter tested every pair of moving averages from 5 days to 300 days on one index, picked the best pair, 37 days and 187 days, and showed a backtest chart that climbed steadily for the whole period.
Marcus notices that nobody would choose 37 and 187 for any reason except that they worked best on that one history, and that is the problem with the video. Settings chosen only because they scored highest on one stretch of past prices are a warning sign.
A backtest applies a trading rule to past prices to see how it would have done. It is the right way to check a rule, and it is also the most common way people fool themselves. Backtests mislead in three main ways: data mining, overfitting and missing costs.
Data mining means searching through many rules on one set of data until some of them look good. The usual statistical test accepts a result if it would happen by chance only one time in twenty. If you test 100 rules that have no predictive power at all, about five will pass on luck alone (100 x 1/20 = 5). If you report only those five, they look like discoveries.
The presenter in Marcus's video tested thousands of pairs and reported only the best one. Data mining doesn't require anyone to be dishonest. It happens whenever someone tries variations until one works and then forgets how many they tried.
Lesson 10.1, What technical analysis claims and what the evidence supports, describes researchers who went back to the rules tested in an earlier study and allowed for data mining. Once they did, the results looked much weaker.
You can guard against data mining in two ways. First, keep a list of every rule you test, the losers as well as the winner. Second, raise the bar in proportion to how many rules you tried. The more rules you tested, the better the winner has to look before you take it seriously.
Overfitting is a close relative of data mining. Every price series contains real patterns, if there are any, plus random noise. A rule overfits when it has enough adjustable settings to be tuned to the quirks of one particular history. It then fits the noise as well as the patterns, and the noise will not repeat.
You can often spot overfitting by its precision. Oddly specific settings such as 37 and 187 are suspect. Round, simple settings such as 50 and 200 days, chosen before anyone looked at the results, are less likely to have been fitted to noise. The number of conditions matters as well: a rule with five conditions, each with its own threshold, has far more room to fit noise than a rule with one condition.
The standard defence is out-of-sample testing, where you design a rule on one part of history and test it once on a separate part you have not touched. With twenty years of data, it works like this:
Split the price history before you start. Use the first twelve years to design the rule and choose its settings. Lock the rule so nothing changes. Test it once on the last eight years. If it works only on the design years, treat it as fitted to that data.
Walk-forward testing is a stricter version. You repeat out-of-sample testing in rolling steps, each time designing on the past and testing on the period that follows.
Two other faults make a backtest look better than it was. Look-ahead bias is using information that was not available at the time. A common case is a rule that acts on a day's closing price at that same close, when the closing price could only be known afterwards.
Survivorship bias, from Lesson 1.7, Read a fund's performance table without being misled, creeps in when a stock data set leaves out delisted companies. Any stock-picking rule tested on that data will look better than it was.
Most backtests in videos and forums show returns before trading costs. Every real trade pays a spread, a commission and some slippage, which you measured in Lessons 5.3 and 5.8. A rule that trades often pays these costs often.
To price costs into a backtest, count the trades per year and multiply by the cost per trade, which is the spread plus the commission, using the minimum commission where it applies. Express the total as a percentage of the account per year, then subtract it from the rule's advantage over buy-and-hold.
Take a rule that beats buy-and-hold by 2 points a year before costs and trades 24 times a year. A cost of 0.15% per trade is reasonable for a large ETF with a low commission. The annual cost is 24 x 0.15%, or about 3.6% a year. After costs, the rule loses about 1.6 points a year against buy-and-hold (2 - 3.6). Most rules that look profitable in forum posts trade far more than 24 times a year, so their costs are higher still.
Small accounts suffer most, because minimum commissions take a bigger share of small trades. Suppose you run a monthly-rebalanced rule in a S$20,000 account, placing four orders of S$2,000 each month. That is 4 x 12 = 48 orders a year. With a minimum commission of S$10, commissions come to 48 x S$10 = S$480. You trade 48 x S$2,000 = S$96,000 in total, and spread costs of 0.15% on that amount are S$144. The total cost is S$480 + S$144 = S$624, and S$624 / S$20,000 is about 3.1% of the account a year. This estimate excludes slippage and currency conversion. The rule would need to beat buy-and-hold by more than 3 points a year just to break even.
A rule deserves some trust only if it passes all four of these tests. It is simple, with few settings, each chosen for a reason before you saw the results. It makes economic sense, so there is a plausible reason it might work, for example investors reacting slowly to news. It survives realistic costs taken from your own broker's fee schedule. And it works on data it was not designed on, in more than one market or period.
Very few rules pass all four, which saves you effort: you can set aside most confident chart claims quickly, and the rare rule that passes deserves a careful test of your own. Lesson 10.8, Test one trading rule with costs on real price history, walks you through that test.
For the activity, open your broker's fee schedule and get the figures you need to price a year of monthly trading in a S$20,000 account: the commission per order, any minimum, platform fees and currency conversion charges.
Using your broker's fees, list what a monthly-rebalanced rule would pay in costs over a year and work it out as a percentage of a S$20,000 account.
Junxiong-WFG Organisation is an authorised representative of AIA Financial Advisers Private Limited (Reg. No. 201715016G).