Better Than Any Testimonial

I was paid a very big compliment by Customer #1 today.

I offered Customer #1 a 50% discount on everything, because they were Customer #1.

And a guy on the other side of the planet took up that offer, and subscribed to backtesting for 12 months for just US$100 (normal price US$200).

A couple of days ago I put up the prices for subscribing to Auto-Trading. You can see them on the Purchase page.

Auto-Trading is subscribed to by the month, and there's a "buy 10 months, get 2 months free" type thing.

I sent an email to Customer #1 to let him know about it, and reminded him of the 50% discount offer.

He declined my offer, and despite my protests that he should just pay half, he went ahead and paid full price for 12 Months of Auto-Trading.

In return, I subscribed him up for 14 months.

It's a beautiful thing when everyone walks away happy.

Dates And Stuff

I received an email saying that the Daily bar for 27 April in Thinking Stuff was the same as the Daily bar for 28 April in Reuters. How could this be so?

Here's the reply:

I believe the issue is because of 2 things.

Firstly, TS shows the date in GMT time. Probably Reuters might use NY time. I'm not sure.

Secondly, Gain Capital starts their day at 17:00 NY time, which is 21:00 GMT in their summer, and 22:00 GMT in their winter. It's a very late time to be starting the day.

I guess a better way to say it, is that Gain Capital *ends* their day at 17:00 NY time, which is the end of the NY session, and so the next day starts then as well.

So when you talk about a Gain Capital *day* marked as "27 April 22:00 GMT", actually that means the time from 27 April 17:00 NY time, to 28 April 16:59 NY time.

So if you think about it like that, then the bar for 28 April actually starts at 27 April 22:00 GMT. On the x-axis of the chart, you can be forgiven for thinking you're looking at the Daily bar for 27 April, but actually you're looking at the Daily bar for 28 April.

It's confusing I know. It's not my fault :-) Oanda starts their day at midnight NY time, so this kind of thing shouldn't be a problem for them.

How To Delete Your Data

Due to a misunderstanding about what "EST" means, unfortunately people already using the Thinking Stuff software need to re-load all the text files from Gain Capital, and re-create all the interval data. See my previous post for more information.

There are 3 ways to clear the old data before you start again. You only need to do one of them:

1. Create a brand new database and start from scratch. But, you would have to also re-enter your user/account/trading system details.

2. Click "File..." > "Price Data" and use the new "Bulk Delete" button. You don't need to click the Search button before using Bulk Delete - just choose the currency, interval, and company for each combination you have created price data for, and click "Bulk Delete" for each of them.

3. Click "File..." > "Data Manager". Click Search to see all the Data Manager entries. Delete all of them. Use the "Bulk..." button to re-create them.

When you delete the Data Manager entries, that deletes the database table containing all the price data. Then re-creating the Data Manager entry re-creates the database table, but which is now empty.

Oh, all of that is only necessary for the Tick data. When you manually create the interval data again, you can check the "Overwrite existing interval price data" option, which will take care of removing the old intervals.

v0.7.3 - General Bug Fixes

Fixed up the File Loader window so it takes NY daylight saving time into account. If you haven't read my previous post, it turns out Americans (well, maybe not all of them) use the term "EST" to refer to New York time. However, not knowing this, I turned to the dictionary to see what EST means, and the dictionary meaning is GMT-5. "Ok, the dates in all those files from Gain Capital are GMT-5," I thought to myself. Wrong! The dates in the files are NY time, which switches from EST to EDT (GMT-4) during their warmer months.

The File Loader window can also recurse subdirectories, and you can specify a filter. For example, a filter of *EUR*USD*.csv will process all EURUSD files in all subdirectories.

The way to create the interval price data has been changed. Before, it used to work out the open/high/low/close for one bar, insert that into the database, work out the open/high/low/close for the bar before that, insert that into the database, and so on and so forth until it came across a bar that already existed. And then stop.

That's great for your bigger timeframes - Monthly, Weekly, Daily, etc. But 5 Minute? 5 Second? Yikes. It took forever.

Thanks to a fantastic guy from Slovenia, it now uses this big SQL statement to insert all the bars in the one hit. It still takes minutes to complete, but not hours like it used to.

But here's the thing - I could only use that new technique for intervals of Hourly and below. Which is lucky, because those were the intervals that were taking forever to finish.

The reason I can't apply the technique to 2 Hourly intervals and upwards, is because of the brokers' start of day. This is the time when the brokers themselves decide when their daily bars start and end. For Gain Capital it's 17:00 NY time (not 17:00 EST as it says on their website), and for Oanda it's midnight (NY time).

I store the prices in the database with a GMT timestamp. So the start of day for Gain Capital changes from 22:00 during the cooler months in the US, to 21:00 during the winter months.

This also means that the intervals from 2 Hourly upwards change their start time. 2 Hourly bars in the American winter start from 22:00, 0:00, 2:00, 4:00, etc. During daylight saving, the 2 Hourly bars start from 21:00, 23:00, 1:00, 3:00, etc.

It takes a far smarter person than me to work out how to include that kind of thing into an SQL statement. (Please contact me if you are indeed a far smarter person than me and can work it out. I'll give you free stuff).

So, what might happen is that it takes longer for the 2 Hourly bars to finish being created, than it does for the 2 Minute intervals. Not much longer though, it's all pretty quick. I created two years worth of absolutely all intervals - 5 Second, 10 Second, 30 Second, 1 Minute, 2 Minute, 5 Minute, 10 Minute, 15 Minute, 30 Minute, 1 Hour, 2 Hour, 3 Hour, 4 Hour, 8 Hour, 12 Hour, Daily, Weekly, Monthly for one currency, and it finished in 1 hour.

If you have loaded files from Gain Capital already, and if you've created the interval price data, my suggestion is to start again. As I said in my previous post, I'm probably affected more than anyone else, and I'm sorry about it, but I think it's for the best. It's going to be far quicker this time around anyway.

There's some other stuff I fixed as well. See the full list of changes in the Change Log.

What you want to do now is go and download it. And then you want to go and purchase.

You want to do that, because this is becoming a fantastic little application.

Don't Hate Me, But...

I'm in the process of changing the File Loader window. This window lets you load all those text files full of price data from Gain Capital.

The change is that you won't need to enter the GMT Offset anymore for the Gain files.

I originally thought all Gain files had dates that were GMT-5 (so you would put -5 in that field). But recently I sought clarification on their forum for what people meant when they said "Eastern Standard Time".

The dictionary meaning of EST is GMT-5. It doesn't change for daylight saving as, by definition, it is the "standard" time for New York.

When New York switches to Daylight Saving time, its timezone switches to EDT, which is GMT-4.

So when someone on their forum asked what timezone the dates in those files were, and someone replied "Eastern Standard Time", I took that to mean all the dates were GMT-5.

That thread is pretty old, and it was the only one I could find regarding this. I took it as gospel.

Then someone asked me about it. I followed up on that thread. It turns out when people were using the term "Eastern Standard Time", what they actually meant was "NY time".

"NY time" is EST (GMT-5) during one half of the year, and EDT (GMT-4) during the other.

It's probably the common usage for "Eastern Standard Time" in the U.S., but I didn't know it.

And that's why I'm updating the File Loader window, so it works out the GMT Offset to use itself for when the files have dates in "NY time".

So why would anyone hate me for that? Because on the help webpage for the File Loader window, I state that "GMT Offset of dates is -5 (minus 5) if using the files from Gain Capital".

So, if someone has gone through and loaded all those files, and then created the interval data, they potentially have slightly incorrect dates in their Thinking Stuff database.

And it's probably better if they went through and did it all again.

However...

It's not as bad as it sounds.

Well, let me first say that I have gone through and loaded absolutely all files from Gain. All 15 currencies. All years back to the year 2000.

And I've created the interval data for most currencies down to Hourly. And for a few of them down to 1-Minute intervals.

The old way that files were loaded, it was taking between 8 and 20 hours per month to load all 15 currencies. So you can see why someone might be annoyed that they have to do it again. Further, creating anything down to Hourly bars was reasonably quick. But anything lower than Hourly, such as the 5-Minute, 1-Minute bars, etc, was painfully slow. Again cause for annoyance that it has to be done again.

Here's the rainbow...

The method to load the tick data from the files has been changed. It now takes between 10 and 20 minutes per month. Much better than 8 to 20 hours.

Further, the method to create the interval data has been changed. It's much faster as well. But that's coming out in the next release.

The end result:

If you haven't started using Thinking Stuff, but only want to load files back as far, but no further than April 2, 2006. Go for it. Use a GMT Offset of -4.

If you haven't started using Thinking Stuff, and want to load years' worth of files, hang back for a couple of days until the next release.

If you have started loading the text files, I don't believe your backtesting results will be affected so much. The dates will be off slightly. Personally I will be starting again though.

It's a pain, I know, and I'm sorry. If it helps, as I said, I have loaded absolutely every file already, and created quite a lot of interval data. Fortunately the next time through will be much, much quicker.

One More Uptrend Thingy

• When the 20-period EMA of the close is above the 21-period EMA of the open.

Anybody else got one?

What's An Uptrend?

Reading through all the educational material I've bought over the years, an Uptrend can be defined as:

• When the close (or low) is above an upward-sloping trend line you've drawn on the chart; or


• When there are successive higher highs and higher lows; or


• When the close (or low) is above a simple moving average (SMA); or


• When the SMA is rising; or


• When a faster SMA is above a slower SMA; or


• Combinations of the above.

Which of these are true? Probably all of the them. Backtest.