Introduction to Backtesting

What is Backtesting in Trading

Backtesting is an essential technique in the trading world and is useful no matter which financial markets you trade. It involves applying your trading rules, or your trading strategy, to past data using backtesting software. This is essentially a historical simulation to show you how well your trading strategy has performed in the past. The objective is to simulate all the signals that your trading strategy would’ve generated in the past and assess their profitability. This allows you to determine the effectiveness of your trading strategy in real-time future trading.

Purpose of Backtesting

Backtesting’s primary role is to help you maximize the probability of positive returns from your trading strategy. It allows you to eliminate unsuccessful strategies by identifying that they lost money in the past. Moreover, backtesting aids in the elimination of fragile strategies by revealing their performance over a wide range of market conditions and parameter values.

The significance of backtesting is evident when faced with the question: “Does backtesting matter if it’s based on past data and future trading conditions could differ?” Consider this – would you prefer to trade a strategy that has shown consistent positive returns over 30 years of backtesting or one that has shown losses over the same period? Clearly, a strategy that lost money in the past is not likely to turn profitable just because you’ve decided to trade it. Backtesting, therefore, provides valuable insights to guide your future trading decisions across all financial markets and timeframes.

Benefits of Backtesting

Backtesting has several benefits that give you an edge in trading. First and foremost, it instils confidence. Knowing your strategy was profitable in the past could indicate potential future positive returns. It also deters you from adopting failing strategies, saving you from potential losses.

Furthermore, backtest simulations aid in understanding your strategy’s dynamics and historical performance. For example, gaining insights about the possible duration of your winners and losers, the average size of winnings and losses, and the frequency of wins and losses prepares you psychologically for future trades. This knowledge and understanding make it much easier to stick to your strategy as you are aware of what to expect. Essentially, backtesting prepares you for the realities of trading, thereby making it easier for you to follow your strategy and maintain confidence.

Limitations of Backtesting

Despite its benefits, backtesting has its limitations – Backtests are not predictive models that show you how rich you will become… The most significant limitation is that they rely on past data. Successful backtests do not guarantee future positive returns. It merely tilts the odds in your favor. The effectiveness of backtesting hinges heavily on its execution – there’s a vast difference between good and bad backtesting.

A common pitfall in backtesting is excessive optimization, where you might test numerous parameter combinations and choose the one that performed the historical performance. For example, if you tested 300,000 combinations of parameter values for a trading strategy and decided to trade the one that worked best over the past 20 years, you’re likely to be disappointed. The strategy might perform poorly in real-time trading because you didn’t validate the system’s stability over time, its robustness across various parameter values, or understand its volatility.

Therefore, good backtesting involves deep interrogation and investigation of your trading strategy. As a trader, the more you understand about how your strategy will work, the more confident you’ll be, and the more likely you are to spot potential problems before they impact your real-time trading negatively.

To conclude, we’ve covered what backtesting in trading is, its purpose, benefits, and limitations. In the next section, we’ll delve into the difference between backtesting and other methods of analysis.

Different Types of Backtesting and Methods of Analysis

In the world of trading, one method has become increasingly pivotal to the success of traders – backtesting. This process involves testing a trading strategy on relevant historical data to ensure its feasibility before risking any capital. While backtesting has its critics, when used correctly, it can provide invaluable insights into market behavior and the performance of your strategy. In this article, we’ll explore the difference between backtesting and other methods of analysis and the importance of each in a comprehensive trading strategy.

Backtesting vs Chart Replay

Chart replay is a common technique used by many new technical traders. It involves going back in time on your charts, observing how your indicators move, identifying entries and exits, and recording this information in a journal or spreadsheet. While this can provide some insight into how a strategy may perform, it’s a time-consuming process and offers a very limited view of market conditions.

In contrast, backtesting using trading software allows you to input your trading rules and apply them over several years of market history across thousands of stocks or other financial instruments. The result is a far larger dataset, providing more information about the strategy’s performance across different market conditions and a larger number of trades. In short, backtesting gives you a wider and more detailed perspective, providing the comprehensive data necessary to refine your strategy.

Paper Trading vs. Backtesting

Another common analysis method is paper trading, a process where you follow your trading rules in real time, logging trades as they occur without risking actual capital. Paper trading has its merits, especially in terms of familiarizing yourself with the dynamics of the market and refining your trading process. However, similar to chart replay, paper trading lacks the comprehensive data necessary to assess a strategy’s profitability fully.

Different market conditions, such as bull markets, bear markets, and sideways markets, can drastically influence a strategy’s effectiveness. These cycles can last several years, making it impractical to gather sufficient paper trading data to assess a strategy’s performance across all market conditions. This is where backtesting’s ability to leverage decades of market data offers an invaluable advantage.

Single Instrument Backtesting vs Portfolio Backtesting

The choice between single instrument backtesting and portfolio backtesting often depends on the type of trader you are and the trading software you use. Some traders prefer focusing on one instrument at a time, an approach that suits single instrument backtesting. But for those who trade stocks or cryptocurrencies, where you might trade across a broad market universe, portfolio backtesting becomes critical.

In portfolio backtesting, you apply your rules to the entire market, picking out trending stocks according to your pre-set rules. Portfolio backtesting helps you assemble a portfolio of stocks or other instruments and evaluate its performance over time, a crucial factor for those trading in stocks or holding a portfolio of stocks.

Conclusion on Different Types of Backtesting and Analysis

While all types of analysis and backtesting methods offer unique insights, it’s clear that systematic backtesting provides a comprehensive understanding that no other method can match. Regardless of whether you’re using chart replay, paper trading, single instrument backtesting, or portfolio backtesting, it’s essential to remember that these are tools in your arsenal, each with their unique strengths and weaknesses.

As a trader, understanding these differences and applying the right tool at the right time is an important part of mastering the market. Backtesting, when done properly, not only equips you with an advantage over other traders but also instills a level of confidence that can’t be achieved through any other method of analysis.

In the next section, we will delve into backtesting different types of instruments, providing you with a comprehensive guide to optimize your trading strategies across various markets.

Backtesting Different Types of Instruments

In this section, I’m going to talk through each of the different asset classes and how backtesting in those asset classes differs. This will save you a lot of time and energy as well as help you avoid a bunch of mistakes that could really hold you back because backtesting each asset class is quite different.

Backtesting Stocks

So let’s first talk about backtesting in stocks since this is probably the most tricky to get right and once you understand backtesting in stocks, you can very easily apply that to many of the other asset classes.

So here’s the challenge in stocks…

There are literally thousands or tens of thousands of different stocks you can choose from. And so when you are backtesting a trading strategy in stocks, you need to have rules that sort through that huge volume of stocks and data to find the trades that you actually want to take.

Narrowing the universe of tradeable stocks using a liquidity filter

The first thing we need to do is filter out the stocks that are basically trash including:

• stocks that have no liquidity
• stocks that have extreme volatility
• stocks that don’t trade very often

Our trading strategies need rules to eliminate all of the garbage that we just don’t want to trade.

One of the best rules to do that is having a liquidity filter. For example, having a rule that requires a stock to turn over $500,000 – $1,000,000 per day on average eliminates most of the junk that’s listed on the market that’s not really tradable. You can trade stocks with lower liquidity than $500,000 turnover per day, but you’ve got to be careful because the liquidity starts to get patchy and your trades will become hard to get in and out of. The stocks also move erratically making your trading results unpredictable.

Taking Stock Market Cycles into account when backtesting stocks

The stock market goes through different phases which include long bull phases, bear markets and sideways markets. Most strategies will only work well in one or two of those different market conditions and typically won’t work across all market conditions. So in stocks, what we do is we have rules that isolate the market cycles that we are most interested in. This is called an index filter or a market regime filter.

For example, if you’re trading long side trend following, you may only want to do that when the broader stockmarket is going up. You generally won’t want to do long side trend trading when the market is in a strong bear market because there is a high chance that those trades will lose money even if the individual stock looks strong. We avoid these market conditions by adding a market regime filter rule to ensure we only take signals at the right time.

What trading rules do you need to backtest stock trading systems?

Once you’ve got the market regime right, then you get down to the trading rules to trade each individual stock. There are tens of thousands of stocks that we can trade on the US market alone, let alone all the other markets around the world (which you should be interested in trading by the way). I believe it is extremely important to diversify into different markets rather than just sticking to your home market (even if your home market is US Stocks).

So how do we find our buy and sell signals for each stock when backtesting?

Your trading system must include objective rules that tell it when to buy and when to sell a stock. These rules are used by your backtesting software to scan the historical data and find every single bar where there was a buy or every single bar where the sell conditions were true. The challenge now becomes that your buy and sell rules actually give you way more trades than you can possibly take.

For example if you trade a breakout trend following strategy then each day you could have hundreds of signals to choose from!

In fact, could be 10, 20, or a 100 times more signals than you could actually take. So the next thing you need is in stocks, is add portfolio construction rules to tell your backtesting software the order of preference to take the trades, how many trades to take and how many shares of each stock to buy / sell.

The position score rule gives your backtester the priority of each trade based on some sort of price behavior. Here are some example position scores for you to consider:

• Prioiritize stocks that have moved the furthest today compared to a 100 days ago
  Close / Ref(Close,-100)
• Prioritize stocks that have the lowest volatility
  ATR(14)/Close
• Prioritize stocks that have the highest volatility first
  Close / ATR(14)
• Prioritize stocks that have the furthest above its 200-day moving average
  Close / MA(C,200)

Those are all examples of position score rankings that you can use in order to prioritize which signals you should take. 

At this point I can almost hear you asking “which position score formula should I use?”… and the answer is of course to Backtest it!

So at this stage of the process we have got:

• our instrument liquidity rules to filter out the trash stocks we don’t want to trade
• our market regime filter to tell us whether we’re in a bull / bear / sideways market
• our buy and sell rules to generate the signals on each stock
• our position scoring rules to rank the trades so the backtest knows which order to take trades

The last step is to define a rule to calculate the position size. The position sizing rules (also known as money management rules) tell us how much capital to put into each trade, and you need to put those position sizing rules into your backtesting software so that it can assemble a portfolio and show you how that portfolio would’ve performed over time. Position sizing is one of the most important aspects of trading and it has a huge impact on your trading system’s performance.

All of those considerations come together to give you a complete stock backtest.

Survivorship Bias – 2 HUGE Challenges you must overcome to backtest stocks correctly

You may have heard of survivorship bias, but what is it and why is it such a big challenge for backtesting?

If you backtest over 30 years or more of history, a lot of stocks come and go. Stocks come into the S&P 500, stocks leave the S&P 500. Also stocks are born, list on the exchange and get acquired or go bankrupt and delist. Both of these are examples of survivorship bias.

There are two main types of survivorship bias:

1. Index constituency survivorship bias
2. Stock delisting survivorship bias.

The index constituency one is by far the most risky and worst to suffer from. If you take today’s S&P 500 stocks, the 500 stocks in the S&P 500 today, and backtest your strategy over the last 30 years on that list of 500 stocks, you’ll get a really positive backtest.

Why?

Because the backtest is cheating!

What you are doing is like getting into a time machine and travelling back 30 years and buying stocks that you know will eventually become part of the S&P 500 index. Thirty years ago many of those stocks could have been penny stocks just starting out and if your backtest buys them because they are CURRENTLY in the index then you are getting an unrealistic advantage in your backtest that you can’t replicate in real life trading (unless you have a time machine) because chances are that stock has gone through a big growth phase in that preceding 30 years to get into the index.

So that’s cheating and you can’t replicate those results in real trading

What you’ve got to do instead is look on the day of the signal and determine day by day in the backtest whether that stock in the S&P 500 index (or any other index you happen to be testing). If it was in the S&P 500 on that day and that is one of your requirements, then take the signal.

If you don’t have the historically accurate index constituency data for every single day of market history, then you can’t backtest or trade strategies that limit their universe to index constituents. You have to backtest the entire universe of stocks (all listed stocks) rather than the index constituents.

I actually prefer to backtest the entire market because often the systems work better, but if you do have the historical index constituency, then you can backtest strategies in stocks that only trade, stocks in the S&P 500 Index or stocks in the ASX 200 index or stocks in the ASX Top 50 or the Nasdaq-100 for example.

If you want to get this historically accurate index constituent data, then you need to subscribe to Norgate Data because that is the only place you can get it >>> Click here to learn more about Norgate Data.

The second type of survivorship bias is stock delisting. If you can get delisted stock data, so you’ve got the currently listed stocks and all the stocks that delisted in the past, it gives you a much bigger database. That’s great because it gives you far more information about how the strategy would’ve worked and you’re not missing any trades that would’ve appeared had you been trading that strategy in the past.

A lot of educators will talk about how you absolutely have to have delisted stock data, and if you don’t, you’re fooling yourself, but delisted stock information is nowhere near as important as index constituent information explained above.

I have backtested most of my strategies with and without delisted stock data. In most cases when you include delisted stock data, the backtested performance of the strategy gets significantly better because it gets more trades and therefore has more chance to profit. 

Stocks get delisted for two primary reasons:

1. They have gone bankrupt
2. They have been acquired by another company

Stocks that go bankrupt generally trend down in stock price for a period of time because of the poor performing underlying business. This downtrend should be enough to cause you to get an exit signal or avoid buying the stock in the first place, so will generally not be holding the stock when it ultimately delists.

Stocks that are acquired generally are acquired at a premium to their recently traded price resulting in a gap up in the stock price. This would cause you to make great returns on that stock if your trading strategy was holding it.

This means that adding delisted stock data generally improves performance of a backtest so it is not that big a deal if you do not have delisted data (plus you can only get delisted stock data for US, ASX and TSX stocks anyway – for other markets you have to do without).

To summarise the survivorship bias issue – If you are backtesting trading systems that only trade stocks within a particular index then you must have historically accurate index constituents. Adding delisted stock data is better than just currently listed stocks, but not absolutely essential if you can’t get it for your markets.

Backtesting Crypto

Backtesting strategies for cryptocurrencies comes with its own set of challenges, primarily due to the limited historical data available for most crypto assets. Unlike stocks with many decades of data, most cryptocurrencies have only a few years of historical trading data. Consequently, accurately evaluating the performance of trading strategies becomes more difficult.

Like stock backtesting, liquidity and market conditions are crucial considerations when backtesting crypto assets. Illiquid and highly volatile cryptocurrencies can be challenging to trade effectively, making focusing on tokens with more substantial liquidity and stability essential.

The primary challenge with crypto backtesting lies in acquiring sufficient historical data for robust evaluation.

Relying on data from a single exchange can lead to a skewed database, as the number of tokens available for analysis will be limited to those listed on that specific platform AND the length of history will be determined by the data the exchange listed that asset.

To address this limitation, it is advisable to use comprehensive datasets provided by reputable sources like Brave New Coin. These datasets amalgamate data from multiple exchanges, allowing traders to access a more extensive historical dataset with a broader range of tokens, leading to more comprehensive and accurate backtests.

For example, let’s say we want to backtest a trading strategy that involves Bitcoin and Ethereum. If we rely solely on data from a single exchange, we might have limited historical data for these tokens, as the exchange may not have listed them from their inception. By using a comprehensive dataset from a provider like Brave New Coin, we can access a more complete history of these tokens’ price movements, enabling a more accurate and reliable backtest.

Considering the dynamic nature of the cryptocurrency market, with new tokens continually emerging, it is crucial to obtain historical data that covers as much of the market’s history as possible. Access to a more comprehensive dataset enables traders to make well-informed decisions when developing and backtesting their trading strategies.

Aside from these crypto specific backtesting issues, all of the same considerations discussed in the stocks section also apply to backtesting crypto trading strategies.

Backtesting Futures

Backtesting futures is quite a different beast than stocks and crypto for two reasons: futures contracts expire and futures are a leveraged instrument.

You can’t just buy one futures contract and hold it for six months or a year because chances are it’s going to expire during that period. Because futures contracts expire and you need to roll from one contract to the next, backtesting with the individual contracts is problematic. So rather than using each individual contract, when you backtest futures you use continuous contracts.

Continuous contracts take all of the different expiry months and splice them together back adjusting as it goes to give you a continuous and smooth price series that mimics what your account would’ve done had you bought that contract, held it to near expiry and enrolled to the next contract and held that one, and then rolled to the next contract and held that one, and so on.

So you need to use those continuous contracts or have trading software that does the contract rolls for you. So that’s generally going to be more specific trading software specific to futures. So software like Trading Blox can do that. Software like AmiBroker, which is what I use, uses continuous contracts rather than managing each individual contract roll.

With futures, the other consideration is whether you are doing portfolio backtesting or single instrument backtesting. Most retail traders probably don’t have enough capital to do run portfolio futures trading strategies. You’ll need several hundred thousand dollars in your account to run a diversified trend following strategy in futures, for instance, because the margin requirements are quite high.

Over time, as the futures exchanges introduce more mini and micro contracts, the contract size becomes less and less of an issue, which is fantastic for traders, but right now there’s still a lot of contracts you can’t trade unless you have a quite large account.

So most futures traders, currently at least, have single instrument strategies because you can trade in and out on a single instrument basis without having a huge account. So if you’re trading single instrument, then obviously you’re going to backtest that way. It’s quite different to backtesting a portfolio of instruments like you do in stocks.

Backtesting futures is quite different to stocks, and you’ll want to make sure that you really learn and understand all of those considerations in detail before launching into futures trading. In particular you need to take into consideration:

1. Contract Rollover: As mentioned above, Futures contracts expire, so your backtesting (and live trading) needs to account for contract rolls at regular intervals. This introduces complexities when backtesting, as historical data must account for these rollovers, so at the very least some additional fees should be built in to account for rollover costs. In addition, the choice of rollover rule which determines when your position rolls from one month to the next can significantly impact results.
2. Leverage: Futures are traded on margin, meaning that traders can leverage their positions. The margin requirement is set by the exchange is is different for each futures contract. This leverage introduces an additional level of risk that must be accounted for during backtesting.
3. Volatility: Futures can be more volatile than stocks (some contracts more than others), especially commodities futures which can be influenced by a range of unpredictable factors such as weather, geopolitical events, or sudden changes in supply and demand.
4. Liquidity: Contrary to what you might think, some futures contracts can be quite illiquid, and liquidity changes as the time to contract expiration approaches. We need to be careful to ensure we are trading and backtesting the most liquid contract months and our positions roll forward to the next month before liquidity in our current contract dries up. Low liquidity can lead to slippage, which is the difference between the expected price of a trade and the price at which the trade is executed.
5. Price Limits: Futures markets often have daily price move limits which are set by the exchange in an attempt to ensure an orderly market and to protect the integrity of the exchange system. Price limits mean the price cannot move more than a predefined amount in a single day. If a price moves up (or down) buy the limit amount then it becomes ‘locked limit’ which means essentially that no further trading can happen that day. This can create gaps in the price series, affecting certain strategies, particularly those that rely on intra-day data.
6. Dividends and Corporate Actions: These exist in stocks, but do not occur in futures contracts.
7. Short Selling: While short selling stocks can be restricted or costly, going short on futures is generally straightforward and is a normal part of futures trading. It is just as easy to go short as it is to go long in futures and there are no restrictions on shorting like there are when shorting a stock. This opens up some different trading strategies for futures which might not be viable for stocks.

Backtesting Forex

Backtesting Forex is very similar to futures except the contracts don’t expire and the levels of available leverage are much higher in Forex. When you trade forex you are simultaneously buying one currency and selling another. To maintain this position you pay a financing cost which is determine by the difference between the interest rates of the two currencies.

Forex traders typically don’t trade a strategy on a diversified portfolio of currencies. Instead they usually trade single instrument strategies. This makes it very quick and easy to backtest. You don’t have contract rolls like you do in futures. You don’t have lots of garbage stocks to filter out like you do in stocks and crypto. So Forex is actually quite simple. You can have trading strategies in Forex that are just literally a couple of rules (since you don’t need all of the filters that you need in stocks and crypto).

Now, getting profitable and consistent, that’s a completely different story!

Here are some of the main differences between backtesting Forex compared to stocks:

1. Market Hours: Forex markets operate 24 hours a day, five days a week. The stock market however only trades for set market hours 5 days a week. This means that forex markets can react to news and events more rapidly than stock markets and there is only a ‘notional’ open and closing time on each daily bar. Many Forex traders trade short term intraday strategies on time frames less than 1 day since the definition of a ‘day’ makes less sense in a 24 hour market, whereas in stocks the open and close of the day are important reference points.
2. Leverage: Forex markets allow for much higher leverage than stocks (and futures). This means that a small change in the price can lead to a large change in the value of the investment. Many Forex traders blow up their accounts because of the high leverage causing wild swings in their account value. This leverage needs to be taken into account when backtesting Forex trading strategies, as the risk and reward profile can be very different compared to unleveraged instruments like stocks and crypto.
3. Currency Pairs: Forex trading involves the simultaneous buying of one currency and selling of another, while stock trading involves buying or selling a single stock. This creates some degree of correlation between many currency pairs because one of side of the pair may be common between multiple currency pairs (eg. AUDUSD, EURUSD, GBPUSD, CHFUSD are all influenced by the strength of the USD). This creates a different dynamic that needs to be accounted for in backtesting and results in portfolio trading strategies that work in stocks generating large drawdowns in the Forex markets.
4. Interest Rates: When trading and backtesting Forex strategies there is a cost to hold the position overnight which is determined by the difference in the interest rates between the two currencies in a forex pair. Many traders trade intraday to avoid this cost which is known as the rollover rate or swap rate. When backtesting Forex trading strategies this rollover rate must be taken into account as the cost can quickly add up and errode system profitability.
5. Trading Costs: In forex trading traders generally do not pay commission as the cost is built into the spread (the difference between the bid and ask prices), while stock trading might involve both a spread and a fixed commission. Additionally, forex trading costs (as measured by the spread) can vary depending on the time of day due to differences in liquidity depending on which financial centres are open during the day. When backtesting Forex strategies we must assume an average spread so that we don’t overestimate the profitability of our forex trading system.
6. Liquidity: Major forex pairs are some of the most liquid markets in the world and so slippage is generally not a problem, however on minor pairs spreads can be quite wide, so Forex traders need to take this into account when backtesting exotic forex pairs.
7. Short Selling: Short selling is more straightforward in forex trading compared to stocks. Like in Futures, shorting in forex is as easy as going long. There are no restrictions or additional costs other than the rollover costs and spreads described above.

So backtesting Forex is probably one of the easier markets to backtest because you typically use single instrument systems and there are no rollovers, we don’t need as many filters because we just select a liquid pair and trade it. However, I think Forex is one of the most competitive markets, and I’ve found certainly that it’s far easier to generate very profitable strategies and stocks and crypto compared to Forex.

Backtesting Options

Backtesting options is very difficult because there is a huge volume of data to deal with as a result of all the different stocks, calls / puts, different strike prices and different expirations. So it’s masses of data!

Backtesting options trading strategies is very problematic because of the huge quantities of data involved.

To assemble all of the data, pull it all into trading software and correctly backtest it all is a huge and very, very difficult task. And to put it bluntly, most trading software is just not up to the task. In fact, I’m yet to find an options backtesting software that I’m really happy with, and even if I did, the data is very expensive because you’ve got to get a huge volume of data from the exchanges.

So backtesting options is very problematic.

Unfortunately, a lot of options traders resort to paper trading and chart replays, and they’re not backtesting properly as we have discussed earlier in this article. The reason I don’t actively trade options is because I can’t effectively backtest the strategies over a long period of time.

The real challenge is that in options, a lot of strategies might look very good, but there’s a big tail risk that can really hurt you. Unless you backtest the options trading strategy over a long period of time you won’t understand the full implication and impact of this tail risk.

There are an increasing number of backtesting platforms online for options trading, and I’m hopeful that in the very near future, those will become competitive and perform as well as the the backtesting software that I use for stocks and crypto, but they’re just not there yet. This is why I’m not backtesting in options right now.

Summary of considerations for backtesting different instruments

Each instrument has a different set of considerations as you have seen. We have discussed the considerations for stocks, crypto, futures, Forex, and options.

I like my traders to start in stocks because it’s the most understandable and it’s the simplest to conceptualize. Once you know that you’ve got to filter out the garbage stocks, use a regime filter, have buy/sell rules, position score rules and position sizing, backtesting stocks is straight forward. It becomes really powerful because you can assemble that portfolio and see how it would’ve performed over time and get really great insights into your trading strategy.

Crypto works amazingly well and is a great way to diversify your stock portfolio. Futures is also great if you have enough capital to do it or if you trade single instrument systems. Forex is very easy to backtest, but very competitive and hard to win in long term. And options just doesn’t allow you to backtest effectively with current technology.

In the next section I am going to discuss how to set up a trading strategy for backtesting. 

Setting Up a Trading Strategy to Backtest

In this section I will share how you can set up your trading strategy to backtest so that you can get an accurate picture of how your trading strategy performs. This process is important no matter what trading platform you use, so I have generalised the process to be useful for you no matter which backtesting platform or backtesting software you use.

Choosing a Time frame & Documenting Trading Rules

The first, and the most important step is to choose your timeframe and document your trading rules that you want to backtest. The time period could be daily, weekly, monthly, or it could be intraday. This doesn’t impact how you backtest provided you have the historical data at the right level of granularity. 

Documenting your trading rules is critical to do BEFORE you start the coding process. A lot of people will crack open the trading software and start coding madly, but this is a mistake. Before you do that, first document your trading rules in terms of pseudocode so you can check the  trading logic is complete.

Pseudocode is a series of logic statements written in plain English which define exactly how the trading system works. This is where you will document the various technical analysis rules and technical indicators you use for your entry and exit rules. 

If you do not have your own trading ideas then at this point you should look at The Trader Success System because it comes with a large number of complete trading systems ready for you to test and apply to your own trading account. Reading a large number of trading books will also help you generate new trading system ideas.

Ensuring your rules are 100% objective

Once you’ve documented your trading strategy rules, the next step is to ensure those rules are 100% objective. One big problem that new systematic traders face when trying to codify their strategy is eliminating the subjectivity from their trading rules.

Traders who use technical analysis often use checklists which can be notoriously subjective and impossible to backtest. All discretion must be removed from the strategy to allow backtesting!

If you can’t point to an indicator (or set of indicators) on a chart and say, this is exactly the conditions that tell me that buy / sell rule is true, then you have subjectivity in your trading rules. This must be eliminated before you can backtest the strategy.

Sometimes you just need to put a stake in the ground to give yourself a place to start:

• “The Stock is trending up” >>> Today’s closing price is above the 200 day simple moving average
• “The stock is not too volatile” >>> The Average True Range is less than 5% of the closing price

If you have got subjective rules, make them objective so that you can then code them and then backtest them correctly.

Fully Define Your Trading Strategy

Once your rules are 100% objective, you must ensure your strategy is fully defined. A fully defined trading strategy documents exactly what you would do under ALL conditions. So there can’t be any edge cases or situations where you have to intervene and manually make a subjective decision – there must be a rule for everything, else you cannot backtest it correctly.

You can only backtest objective trading rules, so think through every single circumstance, and make sure that the rules cover it. Here are some questions to help you work through this:

• How will you prioritize all the different buy signals?
• How many buy signals will take in any one day?
• When should the system stop taking buy signals?
• How many open positions will you hold at any one time?
• How will you calculate the position size for each trade?
• Where’s the stop loss? Is it in market / next day / next bar on open?
• Where’s the trailing stop? Is it in market / next bar on open?
• Is there a profit target? Is it in market / next bar on open?
• What other conditions under which will I exit?
• Is there any panic exit?
• If the index starts collapsing, is there an index exit?
• And so on…

Make sure every decision you have to make for that trading strategy is fully documented and codified. Work through all of the different scenarios, and make sure your rules cover all of them. Once you’ve fully defined your strategy, it’s time to code it.

Code and Error Check Trading Strategy Logic

Now is the time to (finally) open up your trading software and convert your pseudo code rules into code. If you’re a member of The Trader Success System, you get a code library and trading system templates as well as a whole portfolio of complete trading strategies. This means system coding becomes a simple copy and paste job into a preformed trading system template.

So it is It’s dead easy for members of The Trader Success System to codify their own strategies and backtest them. If you’re not a member of the Trader Success System, click the link to join, or, you’re going to have to learn how to code and master that yourself.

The best approach is to convert your pseudocode to code one rule at a time, then make sure you rigorously error check that code as you go. It is far easier to debug as you go rather than trying to get everything right in one go and debug everything at once.

Most trading software like Amibroker has code testing, syntax testing functionality. So you can press the AFL syntax verifier and it’ll tell you if you’ve got any syntax errors. But a syntax error is not the only error you’ve got to look out for. You’ve also got to look out for logic errors and make sure that the code is executing correctly as we discuss below.

Check Backtest Executes Correctly

Once you’ve fully codified your strategy and run your first backtest, you then need to look at several examples of each different type of trade your system generates to ensure they are all executing correctly. For example you need to look at the entry date /  price and the exit date / price for all of the following trade types to ensure on the charts that the backtest is behaving exactly as it should:

• Trades that hit their stop loss
• Trades that exit based on an indicator exit rule
• Trades that hit a trailing stop
• Trades that hit a profit target
• Trades that hit a time stop
• Plus any other combinations that your system generates

You need to check several examples of each so that you can be sure that the signal to buy and sell was generated on the correct bar, and then that was executed on the correct bar at the correct price. You do not need to check every single trade in the database, but you do need to check several examples of every type of trade outcome to ensure they are all executing correctly.

Once you’ve done that, you are now ready to backtest your trading strategy and really start to investigate performance. But you’ve got to do all of this work first and really make sure it’s coded correctly, otherwise any performance analysis you do will be flawed.

Spending time getting this right up front will improve your confidence and eliminate time wasted finding mistakes late in the backtesting process. Once you have done all of this you can go on to optimization, performance improvement, trying different ideas and fine-tuning your trading strategy.

You’ve done all that, it’s time to move on and backtest and optimize your trading strategy, to try and improve the performance so that you can make the most profit possible out of your trading.

Common Backtesting Mistakes

In this section I want to share six common mistakes traders make when backtesting their strategies. Trading mistakes have the potential to lose you huge amounts of money. So if you don’t avoid these then you’re going to have problems in your live account. 

Backtesting Mistake 1: Ignoring Slippage and Commissions

I can’t tell you how many times I’ve looked at backtest reports from signal services, from system vendors, from website selling courses that don’t include slippage and commissions in their backtest results. This is just criminal because it’s basically outright deceptive.

Publishing backtest results that don’t include slippage and commissions is deceptive and highly misleading!

These costs should be factored into your backtests because slippage and commissions are a real cost of trading and you cannot trade without them. Yes there are some brokers in some markets let you trade commission free, but they don’t allow you to trade slippage free!

There’s always a spread and you need to take into account the fact that you may not always get the price that you really want. Slippage and commission are extremely important and can ve very high costs compared to your systems net profit, especially for short term trading strategies.

What is the impact if you don’t include slippage and commission in your backtest results?

For a short term high frequency trading system that has a very fine edge, the impact can be catastrophic. If you have a mean reversion system, for example, and the average profit per trade is 0.5% and you get 0.25% of slippage on the entry and the exit on average, then your system is not profitable. It’s break even at best!

The impact of slippage and commission is most significant for short term, small edge or low expectancy strategies. For example strategies like mean reversion, where you’re trading many, many signals and they’re very short term moves. So you get a small positive gain when you win, and hopefully a small loss when you lose. Slippage and commission can really destroy the profitability of those systems if you are not careful or if you trade illiquid stocks.

If you’re looking at a system to buy or to get signals from and they don’t adequately account for slippage and commission, then you should run the other way!

Slippage and commission is a real and significant cost of trading, even though commission nowadays is very small. When I started trading, I was paying $40 to buy and $40 to sell. It was unbelievable, so much harder to make money, but now you can get a couple of bucks or a 0.08% commission and there’s even commission free.

But what happens if you ignore slippage and commission?

Ignoring slippage and commission makes short term, low edge strategies look relatively much more attractive. As a result you will be drawn to strategies which are shorter term, higher frequency, small profit per trade, and if you trade those with real money, then you will incur some slippage and commission and the profitability will very quickly fall apart and you will lose money!

With longer term systems like a long-term trend following system, it doesn’t make as much of a difference and is less likely to destroy the system.