Importing/Exporting

This tutorial starts from a brand new settings file, loads a text file of historical tick prices, creates price bars from those ticks, and charts the price bars. The file in question was obtained from Gain Capital, but the theory is the same for text files from any source.

Note, however, that the file we used in this particular case was an older one. Gain Capital's newer files are slightly different. See the settings to use on the File Loader window.

This tutorial assumes:

1. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
2. You downloaded some text files containing historical prices.

This tutorial does these things:

1. Loads the historical tick prices from the files.
2. Creates 10-Minute bars from the ticks.
3. Creates Hourly bars from the 10-Minute.
4. Charts the 10-Minute bars.

A very similar process is described (in words) in this post.

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

The files are available at http://freeserv.dukascopy.com/exp/. Also www.dukascopy.com/swiss/english/data_feed/csv_data_export/.

But you can't start the downloading of all the files and go and do other things - you have to export the data you want one-by-one. Scroll about 4/5 of the way down to get to the currencies available. They have |f| next to them. You can choose any format you like, but make sure you choose "Windows" in the "End line" field.

The good part is that they have data going back to 1986 for some currencies.

DukasCopy provides indicative data, which has no spread. During backtesting, make sure you add on a few pips in the "Commission" field to simulate the spread, otherwise your backtesting results will be overly optimistic.

The Currency Secrets website has covered the issue of indicative data in some depth, so I point you to their pages:

• Beware Of indicative data
• Is indicative data really *that* bad?

p.s. I think their conclusion is that as long as you add the spread, it should be ok.

I guess it's always better to use the prices from the broker you are going to use, if possible.

Once they are finished downloading, follow the rest of the instructions to load text files. DukasCopy historical prices are already in price bar format though. So you don't need to do the steps to create price bars, unless you want to create other price bars not provided by DukasCopy.
 

The method described below works the same for any symbol/interval/broker. We will use the example of creating EUR/USD Hourly and Daily price bars from historical tick prices provided by Gain Capital.

These steps assume you have already downloaded the text files containing the historical prices to your computer. See here for instructions for:

• Gain Capital
• Oanda
• DukasCopy
   

Create the Data Manager entries

Here's a good explanation of what Data Manager entries do.
 

1. Create a Data Manager entry to store the tick prices

1a. In ATM, click "Price Data" > "Price data configuration"
1b. Click "Add"
1c. In the top box, choose EUR/USD, Tick, Gain Capital
1d. In the bottom box, choose the bottom option - "Just set up (or keep)..."
1e. Click OK.
 

2. Create a Data Manager entry to store the Hourly bars

2a. (Should still have the Data Manager Search window open).  Click "Add"
2b. In the top box, choose EUR/USD, Hourly, Gain Capital
2c. In the bottom box, choose the second option - "...shorter time-period..."
2d. Choose "EUR/USD, Tick, Gain Capital" from the list.
2e. Click OK.
 

3. Create a Data Manager entry to store the Daily bars

3a. (Should still have the Data Manager Search window open).  Click "Add"
3b. In the top box, choose EUR/USD, Daily, Gain Capital
3c. In the bottom box, choose the second option - "...shorter time-period..."
3d. Now you have a choice - you can either choose "EUR/USD, Tick, Gain Capital" from the list, just like you did with the Hourly bars. That would mean the Daily bars are also made from the tick prices. Or, you can choose "EUR/USD, Hourly, Gain Capital", to have the Daily bars made from the Hourly. It's up to your preference, but the second option would be far quicker.
3e. Click OK.
 

Side note #1:

Each Data Manager entry creates a file on your computer which will store the prices. If you look in your file system you should now see 3 new files: gain_32_tk.tsphd, gain_32_1h.tsphd, and gain_32_1d.tsphd, respectively.
 

Side note #2:

The number 32 is what we call the "FIP ID", or "Financial Product ID". It's a number only useful within the Thinking Stuff world. 32 equates to the currency with symbol "EUR/USD".
 
Now to get the price history from the text files into the ATM files. To do this, we first create a "file load task", then process the task.
 

Load the files

 
4. Create the file load task

4a. Click "Price Data" > "Price history file loader"
4b. Files to load - browse for and select the file(s) to load. Recommend to just choose 1 file if this is your first time.
4c. Fill in the "File format" and "Dates in the file(s)" fields as per these standard settings for your broker.
4d. Price data to be loaded - EUR/USD, Tick, Gain Capital
4e. Click "Add To List"
 

5. Process the task

5a. Still on the File Loader window, go to "The Task List" tab
5b. Click "Start Loading"
5c. Wait.
 

6. Confirm the task worked

6a. Go to the "Completed List" tab.
6b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
6c. Back to the Data Manager Search window.
6d. Right-click on the "EUR/USD, Tick, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
6e. The message that shows up should display a count greater than zero.
 
So now we have the tick prices. What comes next is creating the Hourly and Daily price bars. This time we create "interval creation tasks", then process the tasks.
 

Create the price bars

 
7. Create the interval creation tasks

7a. (Should still have the Data Manager Search window open). Click (normal left-click) on the "EUR/USD, Hourly, Gain Capital" entry to highlight it.
7b. At the bottom of that window, go to the "Bulk Price Bar Creation" tab.
7c. Click "Add To List"
7d. Click on "EUR/USD, Daily, Gain Capital" to highlight it.
7e. Click "Add To List"
 

8. Process the tasks

8a. Still on the Data Manager Search window, go to "The Task List" tab
8b. Click "Start Creating Price Bars"
8c. Wait.
 

9. Confirm the tasks worked

9a. Go to the "Completed List" tab.
9b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
9c. Back to the "Price Data Configurations" tab.
9d. Right-click on the "EUR/USD, Hourly, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
9e. The message that shows up should display a count greater than zero. Same for the Daily one.
 

Cleanse the prices

Not all text files are created equal. Some have minor errors which can throw out your backtesting results. The easiest way to check for these errors is to view the chart. Ridiculous prices are immediately obvious.
 

10. View the chart

10a. Click "Price Data" > "View charts"
10b. Fill in the search criteria.
10c. Click "Chart"
 

11. Cleanse the data

11a. If the chart looks fine then you're all done.
11b. If there are extreme spikes in the chart then it's possible there is some erroneous data in the text files. Follow these steps to perform the cleanse.
 

[Note: these instructions for Oanda were written a couple of years ago. If you have more recent experience of obtaining their historical prices we would appreciate an update].

Oanda, through the FXTicks tool in their FXLabs section, offers tick data going back to January 1st, 2004. This is for five currencies only though - EUR/USD, GBP/USD, USD/CAD, USD/CHF, USD/JPY.

From the website:

In order to qualify for this service, you must have an FXTrade account with a balance of 1000 USD or more. Data may be ordered as far back as January 1, 2004, and as recently as two (2) months ago.

Steps to obtain the files:

1. Go to Oanda to order the files.
2. Fill in the form on that webpage. ATM can handle ASCII Text, CSV, and Pipe-delimited.
3. Repeat step 2 for all currencies that you want the data for.
4. Wait up to 2 weeks for Oanda to process your request. When they notify you, they will provide you with a website link to click.
5. Click the link and download the file. Whether you use downTHEMall! or not is up to you.

Now you have the files downloaded to your computer. But they are huge! Seriously. They are going to be more manageable if you split them into smaller files. That's where Text File Cleaver comes in.

Start Text File Cleaver, select the file to cleave, and select the directory to put the small files in. Click the "Go!" button. Repeat that for each big file you downloaded.

Then follow the rest of the instructions to load text files and create price bars. But on the File Loader window, select the smaller files to load instead of the big file.
 

*.tsatm - settings file

*.tsphd - price history file

*.tssys - exported trading system

*.tsiba - exported alert

*.tsind - exported custom indicator

*.tscht - exported saved chart
 

The File format and Dates in the file(s) fields on the File Loader window have standard settings based on where you got the files from. (You could also just look in the file).

By "file formats", I mean how the price data is arranged in the text file. As an example, here's how Gain Capital price data looks like:

277248713,EUR/USD,2006-11-05 17:00:02,1.271600,1.271900,D

Comparing that with one from an Oanda text file, you can see they are completely different:

13/02/04 08:36:34,1.280600,1.280800
 

Gain Capital

 
Older files up to and including 2009/05 May/Week3

Order of fields - ID, Currency, Data, Bid, Ask, "D"
Delimiter - Comma
Ignore first row - No
Date format - yyyy-MM-dd HH:mm:ss
Timezone - New York (GMT-5/-4)
 

Files from 2009/05 May/Week4 up to and including 2009/11 November/Week3

Order of fields - ID, Currency, Data, Bid, Ask, "D"
Delimiter - Comma
Ignore first row - Yes
Date format - yyyy-MM-dd HH:mm:ss
Timezone - New York (GMT-5/-4)
 

Newer files from 2009/11 November/Week4 to present

Order of fields - ID, "D", Currency, Data, Bid, Ask
Delimiter - Comma
Ignore first row - Yes
Date format - yyyy-MM-dd HH:mm:ss
Timezone - New York (GMT-5/-4)
 

Oanda

Order of fields - Date, Bid, Ask
Delimiter - depends on what you requested from Oanda. XML files cannot be used.
Ignore first row - check the file
Date format - dd/MM/yy HH:mm:ss (note only 2-digit year)
Timezone - Greenwich Mean Time (GMT)
 

DukasCopy

Order of fields - Date, Time, Volume, Open, Close, Min, Max
Delimiter - depends on what you requested from DukasCopy
Ignore first row - check the file
Date format - can't remember - you should check the file
Timezone - Greenwich Mean Time (GMT)

Something to be aware of with these files is that there is no spread included. Therefore ensure you use a few pips of commission when backtesting, to try and mimick the spread. If you fail to do so, you're backtesting results will be overly fantastic.
 

ATM price history files (*.tsphd)

Order of fields - ATM Price History File (*.tsphd)
Delimiter - choose any
Date format - choose any
Timezone - choose any

The File Loader has some mandatory fields that aren't actually mandatory in this case. Just choose any value to get past the mandatory field check.

As per this wiki entry, the price history files have very specific names, e.g. fxcm_23_1d.tsphd. The ones being used by ATM (i.e. the ones which correspond to Data Manager entries) cannot be renamed at all. But if you are wanting to merge a price history file with another, most likely it means you got one from somewhere or someone else. This file you can rename, if you are going to use the File Loader window to merge it. But make sure you keep the file prefix intact. That is, add another dot and type after it, e.g. fxcm_23_1d.temp.tsphd or fxcm_23_1d.copy.tsphd.
 

Text files exported by ATM v3+

Order of fields:

• for files containing tick prices - Date, Utc, FIP ID, Symbol, Bid, Ask, Volume
• for files containing price bars - Date, Utc, FIP ID, Symbol, Bid OHLC, Ask OHLC, Volume

Delimiter - depends on what the setting was when the file was exported.
Ignore first row - check the file
Date format - yyyy-MM-dd HH:mm:ss
Timezone - choose any

As the date and Utc offset is written into these files, the timezone field is not actually required. So just choose any value to get past the mandatory field check.
 

Text files exported by ATM pre-v3

Order of fields:

• for files containing tick prices - Date, Currency, Bid, Ask
• for files containing price bars - Date, Currency, Bid OHLC, Ask OHLC

Delimiter - depends on what the setting was when the file was exported.
Ignore first row - check the file
Date format - yyyy-MM-dd HH:mm:ss
Timezone - Greenwich Mean Time (GMT)
 

Other

ATM supports just about any date format, and any 1-character delimiter. So if you have a text file from somewhere else, all you have to do is somehow get the order of fields to match one of the available options.

If that's beyond your technical prowess, or you know of another popular file format that you think we should support, you can add a post to the feature request forum.
 

There is the Import/Export window, and that let's you import/export trading systems, alerts, saved charts, and custom indicators.

When exporting, your information is saved as an ordinary text file. You can read the contents by opening the file in Notepad, Wordpad, Microsoft Word, or whatever other program you normally use for text files. Like this:

If you add a password when you export the file, most of it will just be garbled text. Like this:

The very basic details are left unencrypted, but everything else is unreadable. The encryption level is pretty strong.

You can then give that file to a friend, sell it, use it as a backup, or use it to copy your settings to a different ATM settings file of your own.

There are some complications with importing/exporting, and most of them centre around the fact that you could be importing something that you yourself exported; or you could be importing something that somebody else exported.

If you are importing your own exported file, you probably want the same accounts to be re-selected for you. The only way to achieve that is to include the account information in the exported file. That is to say, these exported text files may contain private information that you don't want other people to see.

Only the description you gave for the account is exported (e.g. "My FXCM demo account") - not the account number. But there's every possibility that you typed the account number as part of that description (e.g. "My FXCM live account #12345678").
 

The easy workaround

If you intend to give your exported file to someone else, the solution is very easy. In ATM make a copy of whatever it is you intend to export. Edit that copied version. Remove any private information. Export the copied version instead.
 

Protecting your secret sauce

Use a password when you do the export. That will encrypt the file. For someone to be able to import your encrypted file, they need that password. But the password doesn't unlock the file completely - it just lets them do the import.

Someone successfully importing an encrypted file still won't be able to see the trading rules, etc (if it's a trading system, and alerts and custom indicators can also be obfuscated like this). The person will be able to backtest and trade with that encrypted system / alert / custom indicator. But they won't be able to see how it works. As in, you can sell your fantastic trading system, other people can trade using that system, but your methods are kept secret.
 

Statuses of trading systems and alerts

All imported trading systems and alerts have their status set to "Disabled". Check all details are correct before enabling.
 

The files that ATM creates are normal files that you can copy, move, rename, backup, and so on.
 

Two Very Important Points

Firstly, make sure that ATM is closed before moving any files. Secondly, price history files (the ones ending in .tsphd) have very specific names. You can copy/move them to different folders, but you can't rename them.
 

Copying an entire setup

The files to copy include 1 settings file, and then zero-to-many price history files. The folder that the price history files are in is shown on the Options tab, in case you forgot where you put them. Full file paths are also shown on the Data Manager Search window.

Copy all those files to the new computer. Start ATM. Go immediately to the Options window. Change the price history file location if necessary (to match where the files are located on the new computer). If you did change it then re-start ATM.
 

Merging settings files

You can export trading systems, alerts, custom indicators, and charts. The exported files are just text files. Copy them to your other computer and then import. All other settings have to be re-entered manually.
 

Merging price history files

While you can export price history to text files and then import on the other computer, there are better ways. In the examples below, we will say that we are copying the file containing prices for EURUSD/Hourly/FXCM. This corresponds to a file with name fxcm_32_1h.tsphd:

1. If on the second computer you don't have a Data Manager entry for that yet, just copy the file straight to the directory where all the other price history files are. Start ATM, make the Data Manager entry for EURUSD/Hourly/FXCM, and ATM will immediately recognise all of the price history in that copied file.
2. If on the second computer you already have a Data Manager entry for that, but haven't gathered any or much price history for it, just delete the Data Manager entry and follow the steps in paragraph #1.
3. If on the second computer you already have a Data Manager entry for that, and have already started gathering price history for it (enough to make you feel sad to just delete it and follow the instructions in paragraph #1), then you can use the File Loader window to merge the two .tsphd files. It will go faster if you put the larger file in the directory with all the others, and merge the smaller file.

   On the File Loader window, select the file to load, set "order of fields" to "ATM Price History file (*.tsphd)", and choose "EURUSD/Hourly/FXCM" as the price data to be loaded. The File Loader will say that the rest of the fields are mandatory, but they're not in this case and you can choose any value just to get past the mandatory fields check.

Just to repeat, when copying files to or from the computers, make sure ATM is closed.
 

Gain Capital files are available to all - no account required.

1. Download the files from http://ratedata.gaincapital.com/ to your computer.
2. They are zip files, so unzip them.

Then follow the rest of the instructions to load text files and create price bars.
 

An easier way to download

Gain Capital gives one file per week per currency. About 20 currencies equals about 80 files per month and about 960 files per year. That's a lot of clicks. downTHEMAll! will make the downloading much easier than doing it one-by-one.

1. Browse to http://ratedata.gaincapital.com using Firefox.
2. Click through to the month of data that you want to download.
3. Click "Tools" > "downTHEMAll!" > "downTHEMall!..." from the main menu (in Firefox).
4. On the downTHEMall! window:
   1. In the Filters area at the bottom, check the "Archives (zip, rar...)" checkbox.
   2. Click the Browse button to select the directory you want to download the files to.
   3. Click the "Start Downloads!" button.