Backtesting

After you've done a backtest and seen the results, how can you determine whether you can rely on those results or not?

Or maybe you might be thinking now, if the results say it made a lot of profit, then why shouldn't you be able to rely on those results?

Firstly, as per the "Managing Risk" article, the next step from backtesting is auto-trading in a demo account. So no real money will be risked, and it won't matter so much if you've made a judgement to use a system based on unreliable information.
 

Unreliable Information?

I used the word "unreliable" there, but perhaps "incomplete" is more appropriate.

What I mean is, just like a survey, you need a decent sample before you can go making conclusions. If 1 person says that they are going to vote for the conservative party, yes, that is a 100% vote for that party. And yes, if the next person is not going to vote for them, now it's 50-50. But if you want to expand the results from asking 2 people to be the expected results for an entire country of people, you just can't do it. And it's the same with backtesting.

If your backtest results show just 2 trades, they might well both be winners. That doesn't mean you should jump to trading real money with that system though. Even a system that wins just 20% of the time needs to have 2 winners out of 10 at some stage. Maybe your backtest just picked up those 2 winning trades.

The question is then, how many trades do you need for a decent sample size? And that's part science / part art. Because there's no definite answer. You just have to use your best judgement. And remember to auto-trade your system in a demo account before risking real money.
 

Let's say all your brokerage accounts are denominated in USD. And let's say you want to backtest the symbol GBP/JPY. With currency, the calculations are done in the "quote" currency, which is the second in the pair (JPY in this case). The problem is converting those profit/loss calculations done in JPY to your home currency of USD. To do that, you need the USD/JPY rate as well.

This can be done while real-time trading (and in fact is done - converting to your account's home currency that is, not necessarily to USD).

But during backtesting it would be an unnecessary pain. You might not have any USD/JPY historical prices, and if you did you might not have enough to cover the entire backtest period.

So, we just do backtesting calculations and reporting using the quote currency. A 100% p.a. profit is good in any currency, and after all it's the annualised percent profit/loss that is important.

This is why the currency of the starting balance changes depending on the symbol you have chosen to backtest.

[I guess you could make a synthetic pair combining GBP/JPY and USD/JPY which simulates always converting GBP/JPY back to USD, and backtest over that synthetic pair instead. But I haven't really thought that through yet].
 

There are some trading systems settings that can't actually be backtested.
 

Holidays

It's because all you know about the price movement in any given bar is the Open, High, Low, and Close. That's it. You have a zero percent chance of knowing exactly what the price did between those extreme points.

So let's say you are using Hourly bars for backtesting. And you have a holiday that goes from 12:15am to 12:45am, because you want to stop trading during a major announcement scheduled for 12:30am.

Auto-trading can handle this - it just shuts off at 12:15am, and starts again at 12:45am. However, during backtesting, ATM would have to say "exit when the time becomes 12:15am". Easy. But what was the price at which it exited? i.e. what was the price at 12:15am? No idea.

ATM knows what the price was at 12:00am (the Open), and at 12:59:59am (the Close), and it knows the extreme points it reached between those two times (the High and Low), but it has no chance of knowing what the price was at a particular point between those times. And so backtesting can't take holidays into consideration.

However, there is the "Dates and Times" custom indicator, which you might be able to use instead of holidays, and which will work the same in backtesting as it will in auto-trading.
 

Frequency

If you think about it, telling a system to start trading at particular times is just the same but different as telling it to stop trading at particular times. So the frequency settings suffer from the same problem as holidays above.

Again, you would have to try to emulate your desired start times by using the "Dates and Times" custom indicator.
 
See also:

• Pessimistic backtesting results

ATM cannot get around the issue of opening and closing trades in the same bar.

All you know about the price movement in any given bar is the Open, High, Low, and Close. That's it. You have a zero percent chance of knowing exactly what the price did between those extreme points.

As you use longer and longer intervals, such as Hourly, Daily, Weekly, etc, you lose more and more information about what the price did during the bar. Assumptions have to be made during backtesting, and the biggest assumption is this: if your entry price and stop loss price both fit within the range of a bar, we have to assume the worst. That is, the price went up far enough to get us in the trade, and then fell back far enough to take out our stop loss.

A so-called "pessimistic trade".

We want to use the results of backtesting to proceed with confidence that our trading system is going to work in the real world. And therefore pessimistic results during backtesting are going to be better for our bank balance than if it gave overly optimistic results. It's better to be pleasantly surprised during real trading that the system work better than you thought, rather than getting a nasty shock.

And so in ATM, whenever one of these pessimistic trades happens, we simply note it in the summary. If the percentage of pessimistic trades compared to the total trades is very high, then the backtesting results can't really be relied upon. If however there are no, or very few, pessimistic trades, then you can be confident with the backtesting results.
 

The Workaround

Just as the longer the interval the less detail we have, so too the shorter the interval the more detail. Edit the backtest and choose a shorter interval to step through. 1-Second bars are the lowest. That's pretty close to working with tick data. You might be satisfied with 1-Minute or 5-Minute intervals though.

But there's a problem with this, too. Time, and space. Certainly, the lower the interval, the more accurate the backtest. But also the lower the interval, the more bars to process. The more bars to process, the longer it takes to finish a backtest.

It also means that you need to create all those 1-Second bars, which adds extra time, and takes extra space on your computer.

At some point there is equilibrium between the accuracy and the time it takes. But is that equilibrium at 1-Second, 5-Minute, or Hourly bars? That all depends on your trading system and your computer's speed. If you want to test with 1-Second bars, just make sure you do a test run with only 3 hours' worth of prices (~10,000 bars) before running a test over weeks of prices.
 

The whole point of backtesting is to reflect the reality of what happened in the past. Getting ATM to open and close trades in the correct spot is our job. But there's work for you to do as well.

Each time you set up a backtesting configuration entry, ask yourself "does this reflect reality?"

During the backtesting you can have ATM buy 1,000,000 units each trade if you like. But how does that reflect reality if your starting balance is actually $1,000? The broker wouldn't let you leverage to that degree.

You can put in a starting balance of $1,000,000 if you like. But how does that help your decision making if your actual starting balance is going to be $1,000?

Also, each broker has different rules for the number of units you can purchase. Gain Capital, for example, has a minimum of 100,000 units, and goes up in 100,000 increments. EFX Group and MB Trading has a minimum of 10,000 units, and goes up in 10,000 increments. Oanda lets you buy any number of units you like - even 1 or 29,397. So the backtesting results should reflect this. If you are planning to trade with Gain Capital, you shouldn't backtest as if the system will be able to buy any number of units. You should specify that the system can only buy units in chunks of 100,000. Because that's like reality.
 

With all the historical price data in place, and with some wonderful backtesting software, it should be pretty easy to come up with some winning strategies.

Actually it's not. And with good reason. Once you start backtesting, you should hopefully find that it's quite difficult to find a trading system that's profitable. Or at least decently profitable.

That's if your backtesting software works correctly.

And if you think about it, doesn't that fit with reality?

We don't have to make things incredibly complex to make money in trading, but then again it can't be so simple either. If everyone could make money buying when the close is above a simple moving average, and selling when it's below, well, everyone would be.

Then there is the fact that your entry rules are not the only thing you need to worry about when experimenting.

Your entry rules interplay with your stop loss management rules. Changing one effects the other. A trade is made up of both a buy and a sell.

Does your system lose a lot of money because your entry rules are saying to buy at the top of peaks, or because your stop loss is set too close to the price action? You'd really have to chart where your entries are, and how your stop loss is moved compared to the price.

Otherwise you might throw away a trading system that was consistently buying at absolute troughs (i.e. at the perfect time to buy), but was never given enough wiggle room.

Then there's the differences in personalities between the different currencies (e.g. EUR/USD vs USD/CAD). And the differences in personalities between the different intervals of the same currency (e.g. 5-Minute EUR/USD vs Daily EUR/USD). That is, you can't just copy your trading system to a different currency or interval and expect it's going to work the same.

You'd need to alter the entry rules or stop loss management rules. And as I said before, when you change one, you'll probably have to change the other.

You also can't copy a trading system that worked for stocks and expect it to do well with currency. It might, but it probably won't.

So even with (a) a lot of historical price data; and (b) a magnificent piece of software :-) it's still not a walk in the park to riches. But those with (a) and (b) have leapfrogged those without in terms of probability of success, as long as those people with both (a) and (b) continue to think for themselves.

Our software is called "Thinking Stuff's ATM", but it will only do exactly what you tell it.
 

Our Suggestion

Diversify. Not diversify like financial advisors want you to diversify - i.e. buy some stocks, some property, some fixed-interest, etc. Leave that for your retirement fund. No, I mean diversify in terms of trading systems. Don't get stuck to one. Looking around various trading blogs you can find multiple strategies, like Trader Mike and NR7, Chairman Maoxian and Trading For Dummies, I understand Trader-X has one, and there'll be others if you look. You'll have to experiment and adjust, but spend some time doing that and you'll increase your probability of success that much more.

There's also a myriad of e-books that you can buy offering "great" trading systems, but I've just about given up on them. If anyone finds one offering a purely mechanical trading system, please let me know.

See also:

• Example trading systems

To clarify what the "current", or "most recently completed" bar is:

1. Look at a chart.
2. Make sure the chart is scrolled all the way to the right.
3. Point at the right-most bar. This bar is still being made. It's not complete.
4. Point at the bar just to the left of the right-most bar. That's the most recently completed bar.

ATM uses the most recently completed bar for its calculations.
 

Bar Offset

The "bar offset" is the number of bars before the current bar. So a bar offset of zero refers to the current bar itself (the default). A bar offset of 1 means the bar before the current bar. A bar offset of 2 means the bar before that. And so on.

When you create a rule and use a bar offset greater than zero, ATM steps back that number of bars and does its calculations as of the situation as it was back then.

The bar offset is most often used when looking for when one line crosses another. This event is often used for trading signals. To check for such a cross, 2 rules are needed: (1) check that line 1 was under line 2; and then (2) check that line 1 is now above line 2.

Rule (1) has to use a bar offset of 1 to check what the situation was 1 bar ago. Rule (2) uses the bar offset of zero to check what the current situation is.
 

The first step is to make sure you understand about rules and values.

the second step is to make sure your trading system has the minimum required rules and values.

Now, most likely you are reading this article because even though you have read and implemented the above, your trading system refuses to make any trades in your account. So now we will focus on the possible causes. The first step is to diagnose where the problem exactly is.

And this is actually quite simple. Just backtest over a decent number of bars - enough so that at least a few trades should be taken. More is better. The backtest results will fall into one of 3 scenarios:

1. No trades are made at all.
2. Trades are made initially, but then stops.
3. Trades are made often, as you expect.

For the first 2 scenarios, the things to check are listed in the corresponding entry for getting a trading system to trade - backtesting.

Once you trading system does exactly as you expect while backtesting, come back here and finish this checklist.
 

Trades normally

If your backtest results show trades taken as you expected, then the problem is not with your trading system, but with some other setting related to connecting to your broker. Check these:

• Do you have an account yet? If no then read this.
• Is the trading system's status set to "Enabled"?
• Are all other settings on the "Auto-trading" tab set correctly? And the account balance is above the minimum setting there?
• With the account(s) you chose to trade in:
  • Make sure your account balance is enough to purchase the number of units that the system wants to buy.
  • Is the account number entered correctly?
  • Is the password for the user who owns that account entered correctly?
  • Have you installed the API for that broker?
• Is there a holiday attached that's preventing new orders?
• Are the settings on the "Frequency" tab set correctly?
• Have you set the frequency to stop after a certain number of checks, and that number has been reached?
• Are you sure that the entry rules have been true at some point after starting auto-trading?
• Scheduled processes on the Command Centre:
  • Have you started the price download and/or creation process? Is it working and new price bars are coming in? You can check this by using the Chart window or Price History Search window. If no then read this.
  • Have you started the auto-trading process?
  • Are the auto-trading tasks being processed without error? You can check this on the "Automated Trading Tasks" >> "Completed List" tab. Scroll to the right-side of the result grid. Anything in the "Error Text" field is bad.
• Did you accidentally set the trading system's "last bars used" dates or "last checked" date into the future? This will pause trading until those dates are reached.
• Is it possible that trades are being opened, but just ATM's trade and order information is out of date? Check your account through your broker's platform instead. If trading is taking place, then it's all good and read this.
   

Graphical representation of the checklist

Here's what it kind of looks like graphically:

If it still doesn't work

If your trading system still refuses to trade, you will need to seek advice in the forum. Give as much information as you can. An exported version of your trading system would be ideal. If you are worried about your privacy, then make a copy of your trading system. Edit the copy, delete the private stuff, and maybe make a generic version of your system that still displays the same problem but doesn't give away your special sauce. Export that one instead.

Screenshots are also good. We've found that glancing through these screenshots, it takes about 20 seconds to diagnose the problem. Without the screenshots or the exported system it takes a lot longer.
 

The first step is to make sure you understand about rules and values.

the second step is to make sure your trading system has the minimum required rules and values.

Now, most likely you are reading this article because even though you have read and implemented the above, your trading system is not opening trades during the backtest. So now we will focus on the possible causes.

Firstly, are you sure that the test has enough price bars in it for the entry rules to be checked? If you have a moving average of period 200 in the rules, then you need at least 200 price bars to backtest with.

Secondly, are you sure that the entry rules can be true at some point during the test?

With that out of the way, now we focus on the results of the backtest. How the system behaved during the backtest will fall into one of 3 scenarios:

1. No trades are made at all.
2. Trades are made initially, but then stops.
3. Trades are made often, as you expect.
    

1. No trades

This is an indication that your trading system is not set up correctly. Try these:

• Are you sure that you have corresponding values for all the rules?
• Are you sure that you have the minimum required rules and values?
• Do you have any rules which will never be true together? E.g. one entry rule says to enter when the close is above the moving average, while another entry says to enter when the close is below the moving average.
• Long trading - make sure the stop loss will work out to be lower than the entry price.
• Long trading - make sure the take profit (if set) works out to be higher than the entry price.
• Short trading - make sure the stop loss will work out to be higher than the entry price.
• Short trading - make sure the take profit (if set) works out to be lower than the entry price.
• Make sure your account balance is enough to purchase the number of units that the system wants to buy.

Once you have fixed any problems, run the backtest again and start the checklist from the beginning.
 

2. Trades initially, but then stops

The first thing to check is your account balance. If the trades that were made ended up in your balance falling and falling, then eventually there won't be enough balance to make any more trades. This is expected behaviour and in fact the only thing that's wrong is that your system is a dud :-)

If that is not the case, then most likely the cause is related to the "No Trading" settings. These settings can interfere with each other so that after a few trades no further trades are possible. Here's an example:

• Allow up to and including 2 long trades in a row.
• There must be 3 long trades in a row before allowing short trades or orders.

Taken together, it means there will be 2 long trades and then nothing more. After those 2 long trades, the first rule disallows any further long trades until a short. And the second rule is still waiting on another long trade before it allows any shorts.

Once you have fixed any problems, run the backtest again and start the checklist from the beginning.
 

3. Trades normally

If your backtest results show trades taken as you expected, then that is problem solved if all you wanted to do was backtest. If however you are reading this article because you are unsuccessfully trying to auto-trade, head to the corresponding entry for getting a trading system to trade - autotrading.
 

If it still doesn't work

If your trading system still refuses to trade, you will need to seek advice in the forum. Give as much information as you can. An exported version of your trading system would be ideal. If you are worried about your privacy, then make a copy of your trading system. Edit the copy, delete the private stuff, and maybe make a generic version of your system that still displays the same problem but doesn't give away your special sauce. Export that one instead.

Screenshots are also good. We've found that glancing through these screenshots, it takes about 20 seconds to diagnose the problem. Without the screenshots or the exported system it takes a lot longer.