Price history

Scheduled process run from the Command Centre. Some run at recurring intervals of your choosing, such as every minute, every 5 minutes, every 30 seconds, every 24 hours, or whatever. Others are "streaming" in that it's more like flicking a light switch on or off.

These scheduled processes do various things. Some download price bars from a broker, some create price bars, one kicks off automated trading, another alerting, another downloads events from the Forex Factory calendar and converts them to holidays. And so on.
 

Scheduled processes for downloading price bars

There are streaming downloads, recurring downloads, special recurring downloads, backfills, and historical prices.

Streaming downloads lets ATM receive prices from the broker as soon as the broker sends them. The brokers are always broadcasting the new prices, and by turning on the streaming process what you're telling ATM to do is log in to that broker and start saving the prices it receives. Mostly brokers offer tick prices in this way, and then you would need to also run the recurring process for that broker that creates price bars.

Recurring downloads lets you grab a few bars at regular intervals. Special recurring downloads target only those price bars that are needed for trading systems that are auto-trading right then at that time (and active alerts).

Backfill refers to trying to get price bars you might have missed, had your computer been turned off, disconnected from the internet, or crashed.

Historical prices are those going back potentially many years.

Not all brokers offer all options. But yet there are all of these scheduled processes for all the brokers. The ones that don't actually work have an explanatory note, telling you what to use instead. If having those that don't work still appear in the list is a bit strange, you can filter the list to only those that can be run.
 

This tutorial starts from a brand new settings file, loads a text file of historical tick prices, creates price bars from those ticks, and charts the price bars. The file in question was obtained from Gain Capital, but the theory is the same for text files from any source.

Note, however, that the file we used in this particular case was an older one. Gain Capital's newer files are slightly different. See the settings to use on the File Loader window.

This tutorial assumes:

1. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
2. You downloaded some text files containing historical prices.

This tutorial does these things:

1. Loads the historical tick prices from the files.
2. Creates 10-Minute bars from the ticks.
3. Creates Hourly bars from the 10-Minute.
4. Charts the 10-Minute bars.

A very similar process is described (in words) in this post.

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

This tutorial starts from a brand new settings file and goes through to downloading historical prices from FXCM.

It assumes:

1. You installed the FXCM bits when you installed ATM.
2. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
3. You used this information to go and open an FXCM demo account, and also you logged in to that account using the FXCM trading platform and got your account number.

In this tutorial:

1. Try to download prices, but realise you need to create a Data Manager entry first.
2. Try to create a Data Manager entry, but realise you need to create a User first.
3. Enter the FXCM demo user and account details.
4. Create the Data Manager entry for EUR/USD, Daily, FXCM.
5. Start the download.

This is the opposite order to that demonstrated in this video tutorial. But actually the method on this current page is more intuitive and also works out to be a bit quicker.

Anyway... let's download some EUR/USD Daily bars...

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

This tutorial starts from a brand new settings file and goes through to downloading historical prices from FXCM.

It assumes:

1. You installed the FXCM bits when you installed ATM.
2. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
3. You used this information to go and open an FXCM demo account, and also you logged in to that account using the FXCM trading platform and got your account number.

This tutorial does these things:

1. Enters the FXCM demo user and account details.
2. Creates the Data Manager entry for EUR/USD, Daily, FXCM.
3. Starts the download.

We, as creators of the software, know the pre-requisites and create them first. However, a normal person just knows that they want to download prices, so they might try that first without entering any of the pre-requisite information. There's a sister video tutorial doing it that way as well.

Ok, so... let's download some EUR/USD Daily bars...

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

If you're looking at a chart you will notice ridiculous ticks straight away. Because the price will be going up and down quite smoothly, and then all of a sudden there will be a price bar that is incredibly out of whack with all the others.

In some of the older Gain Capital files especially, you can get negative prices, and prices that are 100 times bigger than those surrounding them.

What you have to do first is find the erroneous ticks. Then delete those ticks. And then re-create the price bars.

The method I use to identify the bad ticks, is to create the Hourly intervals from those ticks, and then chart the Hourly bars. If the chart flows nicely, then there is nothing to do. If however there are massive spikes in either direction, then there is a bad tick (at least one, maybe more) within that hour.

Make sure you do this for the bid prices, and then do it again for the ask prices. Sometimes one can be fine and the other has a problem.

Now that you've narrowed down the location of the bad ticks to within an hour, you can use the Price History Search window ("Price data..." > "Price bars, ticks") in order to search for and delete those ticks.

Enter the search criteria. Narrow the from and to date fields so that only that hour's ticks are retrieved. Ensure that the "# of rows" setting is enough to retrieve all ticks within that hour.

Once you have all the ticks for that hour in the result grid, click the "Bid" column header. That will sort the results by that column. The bad ticks will either be at the very start or very end of the results. Select the ticks that are obviously bad (negative numbers, stupidly high or low values compared to the others), and delete them. And do the same for the ask column.

Now the bad ticks are gone, you have to recreate the Hourly intervals and repeat the process. And once all the bad ticks are gone, then you can go about creating the other intervals you want to backtest with also.

The files from Gain Capital seemed to become cleaner as time went on. So the older the file, the more likely it is that some bad ticks will be present. Personally, we aren't going to use any of the data before January 2004.
 

The files are available at http://freeserv.dukascopy.com/exp/. Also www.dukascopy.com/swiss/english/data_feed/csv_data_export/.

But you can't start the downloading of all the files and go and do other things - you have to export the data you want one-by-one. Scroll about 4/5 of the way down to get to the currencies available. They have |f| next to them. You can choose any format you like, but make sure you choose "Windows" in the "End line" field.

The good part is that they have data going back to 1986 for some currencies.

DukasCopy provides indicative data, which has no spread. During backtesting, make sure you add on a few pips in the "Commission" field to simulate the spread, otherwise your backtesting results will be overly optimistic.

The Currency Secrets website has covered the issue of indicative data in some depth, so I point you to their pages:

• Beware Of indicative data
• Is indicative data really *that* bad?

p.s. I think their conclusion is that as long as you add the spread, it should be ok.

I guess it's always better to use the prices from the broker you are going to use, if possible.

Once they are finished downloading, follow the rest of the instructions to load text files. DukasCopy historical prices are already in price bar format though. So you don't need to do the steps to create price bars, unless you want to create other price bars not provided by DukasCopy.
 

The method described below works the same for any symbol/interval/broker. We will use the example of creating EUR/USD Hourly and Daily price bars from historical tick prices provided by Gain Capital.

These steps assume you have already downloaded the text files containing the historical prices to your computer. See here for instructions for:

• Gain Capital
• Oanda
• DukasCopy
   

Create the Data Manager entries

Here's a good explanation of what Data Manager entries do.
 

1. Create a Data Manager entry to store the tick prices

1a. In ATM, click "Price Data" > "Price data configuration"
1b. Click "Add"
1c. In the top box, choose EUR/USD, Tick, Gain Capital
1d. In the bottom box, choose the bottom option - "Just set up (or keep)..."
1e. Click OK.
 

2. Create a Data Manager entry to store the Hourly bars

2a. (Should still have the Data Manager Search window open).  Click "Add"
2b. In the top box, choose EUR/USD, Hourly, Gain Capital
2c. In the bottom box, choose the second option - "...shorter time-period..."
2d. Choose "EUR/USD, Tick, Gain Capital" from the list.
2e. Click OK.
 

3. Create a Data Manager entry to store the Daily bars

3a. (Should still have the Data Manager Search window open).  Click "Add"
3b. In the top box, choose EUR/USD, Daily, Gain Capital
3c. In the bottom box, choose the second option - "...shorter time-period..."
3d. Now you have a choice - you can either choose "EUR/USD, Tick, Gain Capital" from the list, just like you did with the Hourly bars. That would mean the Daily bars are also made from the tick prices. Or, you can choose "EUR/USD, Hourly, Gain Capital", to have the Daily bars made from the Hourly. It's up to your preference, but the second option would be far quicker.
3e. Click OK.
 

Side note #1:

Each Data Manager entry creates a file on your computer which will store the prices. If you look in your file system you should now see 3 new files: gain_32_tk.tsphd, gain_32_1h.tsphd, and gain_32_1d.tsphd, respectively.
 

Side note #2:

The number 32 is what we call the "FIP ID", or "Financial Product ID". It's a number only useful within the Thinking Stuff world. 32 equates to the currency with symbol "EUR/USD".
 
Now to get the price history from the text files into the ATM files. To do this, we first create a "file load task", then process the task.
 

Load the files

 
4. Create the file load task

4a. Click "Price Data" > "Price history file loader"
4b. Files to load - browse for and select the file(s) to load. Recommend to just choose 1 file if this is your first time.
4c. Fill in the "File format" and "Dates in the file(s)" fields as per these standard settings for your broker.
4d. Price data to be loaded - EUR/USD, Tick, Gain Capital
4e. Click "Add To List"
 

5. Process the task

5a. Still on the File Loader window, go to "The Task List" tab
5b. Click "Start Loading"
5c. Wait.
 

6. Confirm the task worked

6a. Go to the "Completed List" tab.
6b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
6c. Back to the Data Manager Search window.
6d. Right-click on the "EUR/USD, Tick, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
6e. The message that shows up should display a count greater than zero.
 
So now we have the tick prices. What comes next is creating the Hourly and Daily price bars. This time we create "interval creation tasks", then process the tasks.
 

Create the price bars

 
7. Create the interval creation tasks

7a. (Should still have the Data Manager Search window open). Click (normal left-click) on the "EUR/USD, Hourly, Gain Capital" entry to highlight it.
7b. At the bottom of that window, go to the "Bulk Price Bar Creation" tab.
7c. Click "Add To List"
7d. Click on "EUR/USD, Daily, Gain Capital" to highlight it.
7e. Click "Add To List"
 

8. Process the tasks

8a. Still on the Data Manager Search window, go to "The Task List" tab
8b. Click "Start Creating Price Bars"
8c. Wait.
 

9. Confirm the tasks worked

9a. Go to the "Completed List" tab.
9b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
9c. Back to the "Price Data Configurations" tab.
9d. Right-click on the "EUR/USD, Hourly, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
9e. The message that shows up should display a count greater than zero. Same for the Daily one.
 

Cleanse the prices

Not all text files are created equal. Some have minor errors which can throw out your backtesting results. The easiest way to check for these errors is to view the chart. Ridiculous prices are immediately obvious.
 

10. View the chart

10a. Click "Price Data" > "View charts"
10b. Fill in the search criteria.
10c. Click "Chart"
 

11. Cleanse the data

11a. If the chart looks fine then you're all done.
11b. If there are extreme spikes in the chart then it's possible there is some erroneous data in the text files. Follow these steps to perform the cleanse.
 

[Note: these instructions for Oanda were written a couple of years ago. If you have more recent experience of obtaining their historical prices we would appreciate an update].

Oanda, through the FXTicks tool in their FXLabs section, offers tick data going back to January 1st, 2004. This is for five currencies only though - EUR/USD, GBP/USD, USD/CAD, USD/CHF, USD/JPY.

From the website:

In order to qualify for this service, you must have an FXTrade account with a balance of 1000 USD or more. Data may be ordered as far back as January 1, 2004, and as recently as two (2) months ago.

Steps to obtain the files:

1. Go to Oanda to order the files.
2. Fill in the form on that webpage. ATM can handle ASCII Text, CSV, and Pipe-delimited.
3. Repeat step 2 for all currencies that you want the data for.
4. Wait up to 2 weeks for Oanda to process your request. When they notify you, they will provide you with a website link to click.
5. Click the link and download the file. Whether you use downTHEMall! or not is up to you.

Now you have the files downloaded to your computer. But they are huge! Seriously. They are going to be more manageable if you split them into smaller files. That's where Text File Cleaver comes in.

Start Text File Cleaver, select the file to cleave, and select the directory to put the small files in. Click the "Go!" button. Repeat that for each big file you downloaded.

Then follow the rest of the instructions to load text files and create price bars. But on the File Loader window, select the smaller files to load instead of the big file.
 

A trick with the dates is to be aware of the start time for that broker's "Day". This affects Daily bars only.

With Gain Capital the start time is 5:00pm New York time. Or more correctly, the end of their day is 4:59:59pm.

So what you might think of as the Daily bar for November 2nd, actually starts at 5:00pm on November 1st.

What it means is if anywhere in ATM (backtesting, charts, view data, etc) you choose a "From" date of "01-Jan-06 00:00:00", the Daily bar starting at "31-Dec-2005 17:00:00" (the bar you think of as that for January 1st, 2006) will not be included. And most likely you did actually want to include that bar.

Similarly, choosing a "To" date of "31-Jan-2006 23:59:59" will include the Daily bar that starts at "31-Jan-2006 17:00:00" (the bar you think of as that for February 1st, 2006). And most likely you did not want to include that bar.

Where the start of the day is midnight GMT, there is no such problem.
 

*.tsatm - settings file

*.tsphd - price history file

*.tssys - exported trading system

*.tsiba - exported alert

*.tsind - exported custom indicator

*.tscht - exported saved chart