Don't Hate Me, But...

I'm in the process of changing the File Loader window. This window lets you load all those text files full of price data from Gain Capital.

The change is that you won't need to enter the GMT Offset anymore for the Gain files.

I originally thought all Gain files had dates that were GMT-5 (so you would put -5 in that field). But recently I sought clarification on their forum for what people meant when they said "Eastern Standard Time".

The dictionary meaning of EST is GMT-5. It doesn't change for daylight saving as, by definition, it is the "standard" time for New York.

When New York switches to Daylight Saving time, its timezone switches to EDT, which is GMT-4.

So when someone on their forum asked what timezone the dates in those files were, and someone replied "Eastern Standard Time", I took that to mean all the dates were GMT-5.

That thread is pretty old, and it was the only one I could find regarding this. I took it as gospel.

Then someone asked me about it. I followed up on that thread. It turns out when people were using the term "Eastern Standard Time", what they actually meant was "NY time".

"NY time" is EST (GMT-5) during one half of the year, and EDT (GMT-4) during the other.

It's probably the common usage for "Eastern Standard Time" in the U.S., but I didn't know it.

And that's why I'm updating the File Loader window, so it works out the GMT Offset to use itself for when the files have dates in "NY time".

So why would anyone hate me for that? Because on the help webpage for the File Loader window, I state that "GMT Offset of dates is -5 (minus 5) if using the files from Gain Capital".

So, if someone has gone through and loaded all those files, and then created the interval data, they potentially have slightly incorrect dates in their Thinking Stuff database.

And it's probably better if they went through and did it all again.

However...

It's not as bad as it sounds.

Well, let me first say that I have gone through and loaded absolutely all files from Gain. All 15 currencies. All years back to the year 2000.

And I've created the interval data for most currencies down to Hourly. And for a few of them down to 1-Minute intervals.

The old way that files were loaded, it was taking between 8 and 20 hours per month to load all 15 currencies. So you can see why someone might be annoyed that they have to do it again. Further, creating anything down to Hourly bars was reasonably quick. But anything lower than Hourly, such as the 5-Minute, 1-Minute bars, etc, was painfully slow. Again cause for annoyance that it has to be done again.

Here's the rainbow...

The method to load the tick data from the files has been changed. It now takes between 10 and 20 minutes per month. Much better than 8 to 20 hours.

Further, the method to create the interval data has been changed. It's much faster as well. But that's coming out in the next release.

The end result:

If you haven't started using Thinking Stuff, but only want to load files back as far, but no further than April 2, 2006. Go for it. Use a GMT Offset of -4.

If you haven't started using Thinking Stuff, and want to load years' worth of files, hang back for a couple of days until the next release.

If you have started loading the text files, I don't believe your backtesting results will be affected so much. The dates will be off slightly. Personally I will be starting again though.

It's a pain, I know, and I'm sorry. If it helps, as I said, I have loaded absolutely every file already, and created quite a lot of interval data. Fortunately the next time through will be much, much quicker.