Search
   Blog Register Login

Sharky's Trading Diary

Current Articles | Categories | Search

50 Gigabytes

Over the last few days I've been loading the many text files of historical price data into my Automated Trading Machine.

All these files are freely available on the web. And here are the instructions on where to get them.

Oanda gave me tick data for 5 currencies going back to January 2004. And Gain Capital gave me tick data for 16 currencies, most of which go back to 2001.

Having loaded each and every file into my database, I can now report how much hard-drive space that takes.

50 Gigabytes.

That's just for the tick data. Now I have to create the interval data. Creating each and every interval for each and every currency will, I dare say, double the amount of hard-drive space required (bringing the total to about 100Gb).

But it's 16 currencies going back 5 years, and 5 currencies going back 2 years. So the total is about 100 years' worth of price data, which roughly equals 1Gb of hard-drive space per year of price data that you should budget for.

posted @ Wednesday, December 06, 2006 3:02 AM by Sharky

Previous Page | Next Page

COMMENTS

How did you get the data out of Oanda?

posted @ Wednesday, December 06, 2006 6:47 AM by Craig


Hi Craig, Oanda gives the files free to anyone with $1000 or more in their Oanda account. So in one sense it is "freely available", and on the other there is some restriction.

Anyone can get the Gain Capital files.

Detailed instructions are at:
http://www.thinkingstuff.com/historical-data.aspx

posted @ Wednesday, December 06, 2006 7:27 AM by Sharky


Ha!, I just closed my Oanda account, crap!, they don't advertise that one do they!
What format does it come it?

posted @ Wednesday, December 06, 2006 7:30 AM by Craig


Oanda offers 4 formats - comma delimited, space delimited, pipe delimited, and XML.

TS can handle the first 3 types.

Even if you don't want to subscribe to any part of TS, you can still use it to load all the text files from Oanda or Gain Capital. It will remove any duplicates.

You can then use TS to create the interval data. Once you've done all that, you can export the interval data into text files of your own, and open them up in Excel or whatever.

But I hope that after using TS to do that, you would find the other parts so useful that you just had to subscribe. Think of it as a Christmas present for yourself :-)

posted @ Wednesday, December 06, 2006 7:40 AM by Sharky


I assume the Oanda data is better than the Gain Capital data, the Gain Capitial data is full of holes and glitches. What is your experence?

posted @ Wednesday, December 06, 2006 8:09 PM by Craig


That's right. I've only created the Hourly bars, but the Oanda charts seem nice and smooth and no gaps.

What I do with the Gain Capital data is create the Hourly bars first. That narrows the erroneous ticks, and they are quite obvious, down to a 1-hour period and makes them easier to find. Delete them, re-create the Hourly bars, repeat until smooth.

There are a couple of gaps in Gain's data, but they seemed to have improved their process in recent times.

posted @ Wednesday, December 06, 2006 11:06 PM by Sharky


So with all this data, and your bitchin backtester, you must have a winning stratergy by now :)

posted @ Thursday, December 07, 2006 1:20 AM by Craig


Give me a chance Craig, I've only just finished creating the Hourly bars :-)

posted @ Thursday, December 07, 2006 4:45 AM by Sharky


Click here to post a comment

First Time?

Quotes

"I have looked at you trading software for some time and I have found it very interesting."
- M.J. (Sweden)

Automated Trading Machine

Featured Articles

Download The Science of Getting Rich free!

Video Tutorials

New Indicators

Blog Categories

Blog Archives

Announcements

Blog

Popular Pages

Membership Membership:
Latest New User Latest: billysy
New Today New Today: 0
New Yesterday New Yesterday: 1
User Count Overall: 346

Members Where Are They:
Visitor [4] : Home
Visitor [1] : Features & Screenshots
Visitor [3] : Blog
Home · FAQ · Download · Purchase · Help · Trading · Blog · Company
Copyright Thinking Stuff Pty Ltd  ·  Contact Us  ·  Disclaimer  ·  Privacy Policy  ·  Sitemap