Oanda

Oanda, through their FXLabs section, has recently started offering tick data going back to January 1st, 2004. This is for five currencies only though - EUR/USD, GBP/USD, USD/CAD, USD/CHF, USD/JPY.

From the website: "In order to qualify for this service, you must have an FXTrade account with a balance of 1000 USD or more. Data may be ordered as far back as January 1, 2004, and as recently as two (2) months ago. Orders may take up to 2 weeks to process."

The files you get are huge, and we recommend using a download manager such as DownThemAll! (see the Gain Capital section for instructions).

Each file contains around 20 million rows of tick price data, and TS will probably not be able to load a big file like that in one go because of a self-imposed 30-minute limit. Instead, please use a file-splitting utility such as Text File Cleaver. Using the default options, Text File Cleaver splits the one big file into a number of smaller files, each containing 65,000 rows. For the file I used, 299 smaller files were produced.

On the File Loader window, just select the smaller files to load instead of the big file.

Although not necessary, there is an order of selecting the files in TS that will result in the #1 file being processed first, the #2 file second, and so on. That is:

1. Scroll to the end of the list.
2. Highlight the very last file.
3. Scroll to the start of the list.
4. Hold down the shift key.
5. Highlight the very first file.
6. Click the Open button, being careful not to highlight any other file as you move the mouse to the button.

The reason for recommending Text File Cleaver (aside from that it's free), is that it keeps each row intact. Other file splitters just split the file after a particular number of characters, which often falls in the middle of a row.


Gain Capital

The files are available at ratedata.gaincapital.com.

There are files there for each currency that Gain Capital allows trading in. Mostly the data goes back to the year 2000.

You should really have an account with Gain Capital before downloading all those files. That's a lot of their free bandwidth you'll be using. Please read my instructions on how to open a Gain Capital demo account.

There are *a lot* of files to download. If you do one at a time you'll probably go insane. I used to do it that way until I found a better method. Using a combination of the free Firefox web browser (an Internet Explorer replacement), and a free Firefox extension called downTHEMall!, you can download a month's worth of files with a couple of clicks.

Alternatively there is a different Firefox extension, called FlashGot, that will do the same thing as downTHEMall!, but it needs you to download and install 3rd-party download manager software along with it (also free). If downTHEMall! doesn't work for you, this will be well worth the trouble. Please visit the FlashGot website for details and instructions, and information on other download managers.

Here are the steps:

1. Download Firefox.
2. Install Firefox.
3. Run Firefox.
4. Browse to the downTHEMall! extension page (using Firefox).
5. Click on the link that says to "Install Now for Windows".
6. When that finishes, close down and restart Firefox.
7. Browse to ratedata.gaincapital.com.
8. Click through to the month of data that you want to download.
9. Click "Tools" > "DownThemAll" > "DownThemAll!..." from the main menu.
10. On the "DownThemAll!" window:
    1. In the Filters area at the bottom, check the "Archives (zip, rar...)" checkbox.
    2. Click the Browse button to select the directory you want to download the files to.
    3. Click the "Start Downloads!" button.
11. Wait 10-60 minutes, depending on connection speed.
12. While you're waiting, look through the other extensions available for Firefox. (Warning: this is addictive).

DukasCopy

The files are available at http://freeserv.dukascopy.com/exp/. Also here.

But you can't start the downloading of all the files and go and do other things - you have to export the data you want one-by-one. Scroll about 4/5 of the way down to get to the currencies available. They have |f| next to them. You can choose any format you like, but make sure you choose "Windows" in the "End line" field.

The good part is that they have data going back to 1986 for some currencies.

DukasCopy provides indicative data, and has no spread. During backtesting, make sure you add on a few pips in the "Commission" field to simulate the spread, otherwise your backtesting results will be overly optimistic.

The Currency Secrets website has covered the issue of indicative data in some depth, so I point you to their pages:

• Beware Of Indicative Data
• Is Indicative Data Really *That* Bad?

p.s. I think their conclusion is that as long as you add the spread, it should be ok.

I guess it's always better to use the prices from the broker you are going to use, if possible.


Getting The Prices Into Thinking Stuff

Now you have all the files on your hard-drive. What you need to do after that is get the prices contained within those files into the Thinking Stuff database.

There are a few Gain Capital files which are formatted differently to the rest. So, firstly, download the replacement files at the bottom of this page.

Next use the "Data Manager" window to set up Thinking Stuff to store the currencies that you want to store. Please budget around 1 Gigabyte of disk space per currency per year, and that's just for the tick data.

Then use the "File Loader" window to load the text files into the database. To open, click "File..." > "File Loader" from the main menu.

The File Loader window will load 1 Month's worth of files (15 currencies) in 10 to 20 minutes. Alternatively, it will load 15 Months' worth of data for 1 currency in the same amount of time.

The dates in the Gain Capital files are in NY time. I'm pretty sure the Oanda dates are all GMT, but I haven't confirmed this yet.


Cleansing The Data

Okay, so once all the tick data is in the Thinking Stuff database, now you need to delete some erroneous ticks that were in the files. (This is true for Gain Capital, and probably isn't needed for Oanda, although we can't guarantee it).

The method I use to identify the bad ticks, is to create the Hourly intervals from those ticks, and then chart the Hourly bars. If the chart flows nicely, then there is nothing to do. If however, there are massive spikes in either direction, then there is a bad tick (at least one, maybe more) within that hour.

Make sure you do this for the bid prices, and then do it again for the ask prices. Sometimes one can be fine and the other has a problem.

Now that you've narrowed down the location of the bad ticks to within an hour, you can use the "View Prices" window in order to search for and delete those ticks.

To open that window, click "File..." > "View Price Data" from the main menu. Then enter the search criteria. Ensure that the "# of rows" setting is enough to retrieve all ticks within that hour.

Once you have all the ticks for that hour in the result grid, click the "Bid" column header. That will sort the results by that column. The bad ticks will either be at the very start or very end of the results. Select the ticks that are obviously bad (negative numbers, stupidly high values), and click delete. And do the same for the ask column.

Now the bad ticks are gone, you have to recreate the Hourly intervals. And once all the bad ticks are gone, then you can go about creating the other intervals you want to backtest with also.

Below are some erroneous ticks from the Gain Capital files. This is probably not an exhaustive list. Please check the charts by yourself as well. Personally, we aren't going to use any of the data before January 2004.

AUDJPY
29-JAN-04 07:48:10, 81.73, 82.67
29-JAN-04 07:48:11, 81.74, 82.67
29-JAN-04 07:48:11, 81.73, 82.67
29-JAN-04 17:12:13, 80.61, 81.50

AUDUSD
Everything before 22-MAY-01 12:00:00
15-JUN-01 15:12:33, 0.5201, 0.5280 (may have been JAN)
15-JUN-01 15:13:25, 0.5200, 0.5279 (may have been JAN)
29-OCT-01 21:13:03, 0.5000, 0.5059
29-OCT-01 21:13:07, 0.5000, 0.5059
29-OCT-01 21:13:08, 0.5005, 0.5059
19-DEC-01 23:25:09, 0.5117, 0.5216
26-DEC-01 16:15:51, 0.5072, 0.5171
26-DEC-01 16:16:47, 0.5079, 0.5178
26-DEC-01 16:27:35, 0.5082, 0.5180
07-JAN-02 22:49:41, 0.5185, 0.5284
09-JAN-02 14:42:13, 0.5232, 0.5331
31-JAN-02 04:27:12, 0.5074, 0.5161
15-APR-02 21:01:25, 0.5318, 0.5415
29-JAN-04 07:48:10, 0.7720, 0.7810
29-JAN-04 07:48:11, 0.7720, 0.7810
29-JAN-04 17:12:13, 0.7602, 0.7696
10-MAR-04 20:10:00, 0.7489, 0.7538
10-MAR-04 20:12:00, 0.7487, 0.7532
24-NOV-04 15:05:05, 39.8192, 39.8197
21-NOV-05 20:12:40, 4.0982, 4.0986

CHFJPY
25-FEB-03 16:45:27, 2.1296, 2.1306
03-JUN-03 08:55:05, 118.64, 118.69
15-JUN-04 00:28:12, 103.47, 103.54

EURAUD
28-JAN-04 13:30:15, 1.6209, 1.6307
28-JAN-04 13:30:16, 1.6211, 1.6307
28-JAN-04 13:30:17, 1.6211, 1.6307
28-JAN-04 13:30:18, 1.6212, 1.6307
28-JAN-04 19:16:57, 1.6151, 1.6248
28-JAN-04 19:17:10, 1.6151, 1.6250
27-JUL-04 00:25:32, 1.9926, 1.9936
23-SEP-04 18:46:50, 1.7366, 1.7376

EURCHF
10-APR-02 15:10:21, 1.4687, 1.4785
17-APR-02 21:01:25, 1.4699, 1.4797
26-MAR-04 21:21:00, 1.5580, 1.0000

EURGBP
01-SEP-04 21:01:24, 1.2363, 1.2368
01-SEP-04 21:01:27, 1.2364, 1.2369
26-MAR-04 21:23:00, 0.6679, 0.6000

EURJPY
20-MAY-04 16:00:19, 67.46, 67.51
20-MAY-04 16:00:19, 67.49, 67.54
20-MAY-04 16:00:20, 67.51, 67.56
13-AUG-04 10:01:42 TO 10:02:42 (16 items), change 151 to 136
17-AUG-04 13:04:27, 67.93, 67.98
13-SEP-04 08:25:55, 67.20, 67.25

EURUSD
26-OCT-00 15:25:53, 0.8201, 0.8272
08-JUN-01 03:32:24, 1965.0001, 1965.0001
12-SEP-01 00:12:15, -0.0190, -0.0001
12-SEP-01 00:12:56, -0.0190, -0.0001

GBPCHF
26-MAR-04 21:28:00, 2.3315, 2.0000

GBPUSD
24-NOV-00 14:57:22, 0.9999, 0.9999
24-NOV-00 14:59:33, 1.3999, 1.4005

NZDJPY
Everything before 06-SEP-06 15:00:00

USDCAD
Everything before 02-JAN-01 09:00:00

USDCHF
04-MAY-01 04:57:12, 1.7327, 1.7733
31-OCT-02 03:01:56, 1.4635, 1.4639
25-NOV-02 00:59:34, 1.4300, 1.4305
26-NOV-02 01:48:48, 1.5364, 1.5369
26-NOV-02 05:01:28, 1.4353, 1.4358
26-NOV-02 05:59:46, 1.4335, 1.4340
12-DEC-02 00:47:44, 1.5124, 1.5129
30-MAR-04 14:59:38, 0.6393, 0.6398
30-MAR-04 14:59:39, 0.6392, 0.6397
30-MAR-04 14:59:40, 0.6392, 0.6397
30-MAR-04 14:59:40, 0.6393, 0.6398
17-MAY-04 15:37:49, 0.6408, 0.6413
11-JAN-05 12:49:31, 1.6748, 1.6753

USDJPY
27-FEB-02 15:43:14, 134.41, 135.40
17-JUN-04 07:46:27, 114.74, 114.78

I've gone through and fixed a lot of the Gain Capital files for you.

Those files that needed fixing are now in one big zip file, about 18Mb, and ready for download. But, you need to register before you can access that file. There's a link to register at the very top of this page. Once you do so, the link to download the file will appear instead of this text. Gyah! One more username and password to remember. I know, I know, I'm sorry.

We have a strict privacy policy, so don't worry about your email being sold or anything like that. But after registering, we might (or might not) contact you in order to ask for suggestions and ways to improve our software and website. (You can of course provide that information at any time though the Contact page).

That's it. No marketing. No spam. You can just delete that email if you want, but it's better for everyone if you speak your mind.