There are some traps you need to be careful of when doing a backtest. Most can be overcome though.
Backtesting, I think, is essential. You've probably heard over and over again that past performance does not necessarily reflect future results. And that's very true - essentially the future prices would need to be exactly the same as the ones you tested it on for you to guarantee the system's performance. Which won't happen. Probably.
But surely you wouldn't trade with real money on a system you knew to be a dog for the last couple of years?
So I guess backtesting lets you filter your list of systems to a just a couple that have been performing well up to now.
This should all tie in with your Equity Curve Analysis. You only trade real money on the systems that meet your criteria for allowing real money to be traded. If you're just starting off, then the only way to know if it meets your criteria is to backtest and found out.
Or you could paper trade every system you have thought of in real time, until one or two of them meet your critiera. But I don't think anyone has the patience for that. So it's off to backtest town for us.
But, and here's the thing, you can do backtesting the wrong way. And when that happens you're in possession of faulty results, and you'll be making your decision to trade a system on faulty information.
Whenever you read a forum post saying they believe backtesting to be a waste of time, that person has done just that. And most of the time they'll go on to explain what it was they did wrong while backtesting. (Yet still fail to realise it was them doing the backtesting incorrectly, not backtesting per se, that was the problem).
Here's a tip - whenever you create a system that has mind-boggling results - results which would make you rich beyond your dreams in about 6 months, and leaves you kind of giddy with excitement - you've got a bug in your backtesting process. For sure. It's happened twice to me. The good part is, after you get over the depression brought about from realising you're not going to be rich in 6 months, you've found your bug and your backtesting process has become that little bit better.
The following articles will outline the common mistakes people make when backtesting, along with some of the things that backtesting just can't do.
[PWB = Problem With Backtesting]
Yes, yes, forgetting about the spread (in currency trading, or forgetting about the commission for traders of other things). Essentially, forgetting about any miscellanous costs of trading.
Simply, if you can make $5 each and every time you trade, you'd become rich very quickly. Unless... it costs you $10 in spread/commission/other each and every time you trade. And sometimes people forget that $10 in spread/commission/other bit.
Here's an example. A while ago I thought I'd stumbled onto the world's most simplest yet profitable trading system. I couldn't believe others hadn't thought of it. I was giddy with excitement.
The system I thought of, on the 5 minute chart, was to set a long entry order at the high of the last 5 bars, with stop loss at the low of the last 5 bars. Each bar move the stop loss to the low of the last 5 bars. Eventually, hopefully, the low of the last 5 bars would rise above the initial entry point, and I'd make a profit. Reverse for shorts.
You are laughing because that's a simple breakout system, and it's been around since the dawn of trading. I know that too... now.
Anyway, I was looking at the EURUSD 5 minute chart of the bid (in Oanda's charts you can chart the bid or ask or the average), and "backtested" over about 2 days worth of data. The end result was faaaantastic: 112 pips profit. 112 pips in that short time! If you trade 1 contract and make 112 pips profit, that equals US$1120. All I could see were dollar signs.
80 trades were executed in those 2 days. The problem I had with the fantastic system that I'd "created" was that I wouldn't have been able to sleep - the system required constant attention as it traded so often. (Hmmm, might need a trading-bot to help me out, I might have thought at the time :-)
And the other problem was that I'd forgotten all about the spread. The prices I worked off were all bid prices. Take the spread into account, which was 1.5 pips at Oanda, times 80 trades = 120 pips. 112 pips profit minus 120 pips spread = -8 pips.
Excitement... giddiness... fading...
Let's say you have only interval data (open, high, low, close) on which to perform your backtesting (i.e. no tick data).
And let's say your system wants to enter long at the high of the most recently closed bar, and it wants to set it's stop loss at the low.
And let's say the following bar has a higher high than the one used for the entry and stop loss calculations, and a lower low. There are three possibilities here:
Firstly, the price may have first risen above the high of the previous bar, thus triggering the long entry. We're in the trade. And then the price dropped below the low of the previous bar, thus trigging the stop loss. We're out of the trade at a loss.
Secondly, the price may have first dropped below the low of the previous bar, thus triggering nothing because the stop loss doesn't come into effect until we're in the trade. Then the price rose above the high of the previous bar, thus triggering the long entry. We're in the trade.
Thirdly, the price may have first dropped below the low of the previous bar, thus triggering nothing because the stop loss doesn't come into effect until we're in the trade. Then the price rose above the high of the previous bar, thus triggering the long entry. We're in the trade. And then the price dropped below the low of the previous bar, thus trigging the stop loss. We're out of the trade at a loss. While unlikely, in fact the price could have rebounded many times between the high and low of the bar during the time the bar was formed - there's no way to know if tick data is not available.
This third scenario forces us to have a policy of once being stopped out of a trade, we wait for the bar to complete before calculating a new entry point (as opposed to using the old entry price calculation to get back into a trade if the price rises to that level once more). That's my policy, anyway. Backtesting would be impossible without it as we would not be able to predict the results in this situation.
Anyway, the first and second scenarios alone highlight a problem with backtesting using interval data. Two completely seperate outcomes are possible. And so, I think there is no other option but to either (a) track both possibilities and show two sets of results at the end of the test; or (b) always assume the worst case - that the price first went in a direction in order to get us in to the trade, and then went in a direction that took out the stop loss.
Feel free to ask questions if I didn't explain that very well.
All that means some valid trading systems cannot be backtested. (These are different to Impossible Trading Systems which can be backtested, but can't actually be traded).
For example, scalping techniques are generally always exiting their trades in the same bar that they entered the trade. Unfortunately this means any scalping techniques have to be backtested using tick data, rather than interval data.
But, if you're looking to enter the scalping business, it's probably worth your while to go through the extra steps necessary to backtest using tick data.
Once you've set your entry price, it's the Ask price that determines if your order is taken up or not. Or if you place a Market Order, you get in at the Ask price.
Once in the trade, the Bid price determines if your Take Profit or Stop Loss is hit. Or if you Exit At Market, you exit at the Bid price.
Once you've set your entry price, it's the Bid price that determines if your order is taken up or not. Or if you place a Market Order, you get in at the Bid price.
Once in the trade, the Ask price determines if your Take Profit or Stop Loss is hit. Or if you Exit At Market, you exit at the Ask price.
That's for currency anyway - with stocks and such the price is the price is the price. But don't forget the commission/brokerage.
I hope everybody knows what a "gap" is.
Well, sometimes your backtesting routine will determine that it needs to set a long entry order at a particular price. And probably your backtesting routine will say "ok, our intended entry price is at this level, did the price go above that point? If so then we're in the trade". Or something like that.
And your backtesting routine will record that it entered long at that price:
21-Nov-2005 18:00:00, Entered long trade at 1.2995.
Problem is though, that just because the price went higher than your intended entry price, it doesn't necessarily mean that you were able to open the trade at that price. Because of gaps.
You can't just assume that you were filled at 1.2995 - you have to check if the open price of the bar was below your intended entry first. If it was, then you can say you entered at 1.2995. If it wasn't, that's where your last gasp entry price comes in - was the open below that? If it was, then you can say you got in at the open price. If the open was above the last gasp, you didn't enter the trade.
What if the price was higher than your last gasp, and then came lower after the open? Did we enter the trade or not? Well, I tend to think that if the open is higher than the last gasp, then we cancel the order. We don't know what's going to happen in the future. And if you enter long while the price is going down, that seems to me to be the wrong thing to do.
The same rule applies for exits, although we don't have a "last gasp" here. The rules say we want to move our stop loss to 1.2995, for a break-even setting. The next bar opens above that price, and drops through it, then we can say we exited at 1.2995. But an open below our stop loss means unfortunately we just have to get out at whatever the open price was. There's no waiting for it to come back. That's praying. Maybe it will and maybe it won't. Stop losses are about risk management, and must be followed.
That's why in a system where you set the stop loss to be exactly 50 pips below the entry, you might still end up with a loss of 55 pips or something.
So that's gaps covered. There's also this thing called slippage. You might like to include a standard slippage of a number of pips just to be on the safe side. You want your backtesting results to be more negative rather than more positive. Slippage is where you want to get in at 1.2995, but your broker fills you at a higher price. Common with stocks and options. Less so with currency, but still possible.
Essentially, whatever your profit/loss was, just deduct the slippage from that and calculate a new profit/loss.
If you create a trading system that buys and sells about 80 times a day, essentially that system is impossible for a single human to trade. Either you would need to automate it, or have a group of people so you could get some sleep at some point.
If you can only guarantee you can be at the computer between the times of 2pm and 4pm, then your backtesting cannot scan data over the entire 24 hours of the day. It must only use data available between 2pm and 4pm.
If your account does not allow hedging (that is, holding both a long and short position in the same currency in the same account), then backtesting a system that hedges is pointless. (Although you should be able to set up two accounts - one for your long trades and one for your short).
A post over at Trader Eyal shows another type of situation where your backtesting might not reflect the reality - he wanted to go short on some stocks, but the broker he uses did not have any of that stock available. This is for stocks, I'm not sure currency would have the same situation, but there it is.
I heard a common mistake when creating a backtesting program, is to use the close price of a bar for decision making during that same bar.
The problem is, until that bar is finished, you don't know what the close price will be. You know it while backtesting because it's historical data. But not in real-time trading, and real-time trading is what backtesting is trying to emulate.
And it might not be so obvious that this is going on. If you're explicitly using the close price, then sure, it's obvious. But if you're using an indicator value that is calculated from the close, you might not realise.
Anything related to the close of the bar, can only be acted upon in the next bar.
Thanks to Trader Eyal for this one.
The holy grail of backtesting is to get it as close as possible to real life trading. Should you have true and accurate tick data, backtesting can be very close to reality. But, most people have just daily data. Sure we know the open, high, low, and close, but assumptions need to be made about what happened during that day. As long as your assumptions are reasonable, and lean more towards the pessimistic than optimistic, then accurate backtesting can still be done.
Inaccurate backtesting is a waste of time. And will probably end up a waste of your money, too.
When we have accurate backtesting, we can start to filter the good trading systems from the bad. And then start trading the good systems in a demo account. Testing your system in a demo account requires no assumptions. Do well there and you should do well in real trading (forgetting for a second about trading psychology and emotions and all that, if you're trading manually).
Our automated trading machine of course takes into consideration the issues highlighted in this series. Download it and give it a shot :-)