v0.7.3 - General Bug Fixes

Fixed up the File Loader window so it takes NY daylight saving time into account. If you haven't read my previous post, it turns out Americans (well, maybe not all of them) use the term "EST" to refer to New York time. However, not knowing this, I turned to the dictionary to see what EST means, and the dictionary meaning is GMT-5. "Ok, the dates in all those files from Gain Capital are GMT-5," I thought to myself. Wrong! The dates in the files are NY time, which switches from EST to EDT (GMT-4) during their warmer months.

The File Loader window can also recurse subdirectories, and you can specify a filter. For example, a filter of *EUR*USD*.csv will process all EURUSD files in all subdirectories.

The way to create the interval price data has been changed. Before, it used to work out the open/high/low/close for one bar, insert that into the database, work out the open/high/low/close for the bar before that, insert that into the database, and so on and so forth until it came across a bar that already existed. And then stop.

That's great for your bigger timeframes - Monthly, Weekly, Daily, etc. But 5 Minute? 5 Second? Yikes. It took forever.

Thanks to a fantastic guy from Slovenia, it now uses this big SQL statement to insert all the bars in the one hit. It still takes minutes to complete, but not hours like it used to.

But here's the thing - I could only use that new technique for intervals of Hourly and below. Which is lucky, because those were the intervals that were taking forever to finish.

The reason I can't apply the technique to 2 Hourly intervals and upwards, is because of the brokers' start of day. This is the time when the brokers themselves decide when their daily bars start and end. For Gain Capital it's 17:00 NY time (not 17:00 EST as it says on their website), and for Oanda it's midnight (NY time).

I store the prices in the database with a GMT timestamp. So the start of day for Gain Capital changes from 22:00 during the cooler months in the US, to 21:00 during the winter months.

This also means that the intervals from 2 Hourly upwards change their start time. 2 Hourly bars in the American winter start from 22:00, 0:00, 2:00, 4:00, etc. During daylight saving, the 2 Hourly bars start from 21:00, 23:00, 1:00, 3:00, etc.

It takes a far smarter person than me to work out how to include that kind of thing into an SQL statement. (Please contact me if you are indeed a far smarter person than me and can work it out. I'll give you free stuff).

So, what might happen is that it takes longer for the 2 Hourly bars to finish being created, than it does for the 2 Minute intervals. Not much longer though, it's all pretty quick. I created two years worth of absolutely all intervals - 5 Second, 10 Second, 30 Second, 1 Minute, 2 Minute, 5 Minute, 10 Minute, 15 Minute, 30 Minute, 1 Hour, 2 Hour, 3 Hour, 4 Hour, 8 Hour, 12 Hour, Daily, Weekly, Monthly for one currency, and it finished in 1 hour.

If you have loaded files from Gain Capital already, and if you've created the interval price data, my suggestion is to start again. As I said in my previous post, I'm probably affected more than anyone else, and I'm sorry about it, but I think it's for the best. It's going to be far quicker this time around anyway.

There's some other stuff I fixed as well. See the full list of changes in the Change Log.

What you want to do now is go and download it. And then you want to go and purchase.

You want to do that, because this is becoming a fantastic little application.