Bit Of A (Long) Status Report

If you read through some of the blog entries here, you'd know that I'm making some software to help me out with my trading. I wanted to come up with trading rules that were purely objective - so mechanical that a computer could do it. And then I decided that if a computer could do it, then a computer should do it. You can read here my thoughts on the advantages and disadvantages of automated trading.

And I thought that I should make it as easy as possible for me to create and edit systems, so as to make experimentation incredibly simple. That meant no coding of systems! None. Just selecting rules from pull-down lists. It takes about 5 or 6 mouse clicks to add a new rule to a system, about 3 or 4 to edit its parameters, and about 2 to remove.

Turned out to be quite a bit involved with making an automated trading-bot. Who would have thought? :-) So it's taken quite some time.

And then I decided that other people might like to benefit from my hard work and use my software as well, in that tried-and-tested fee-for-service kind of way.

So, for anyone interested, if there is anyone, I wanted to clarify when the software might be available, and what it will and will not do when you get your hands on a copy. What's written here takes precedence over anything else you may have read on this site.

Here are the tasks remaining:

Unfortunately the thing with software development is that an hour worked doesn't necessarily amount to the same amount of work being completed. Some of the tasks in that chart will take 2 days, and some will take 2 hours. Sometimes completing a 2 day task will leave me with several 2 hour tasks, thus actually making the chart of tasks look worse than it really is. So it's not the best of tracking methodologies, but it's just me working on this - it's not like I have a large team of developers. Or a budget.

Anyway, as that straight line I drew suggests, I'm hoping to have something ready by Christmas. A nice present to myself after many months of hard work.

I've split the functionality out into modules, so here's what it does described in a modular-type way:

Brokers
There are a few brokers that provide APIs (interfaces for software to make trades on their platforms). And here's the funny, funny thing - each and every one utilises a different technology in order to get the job done. Oanda provides a DLL, Gain Capital utilise webservices, Interactive Brokers provides an OCX.

The Gain Capital interface will be done first, as they provide the API free. This will be complete on the initial release.

After the initial release, I will get on to finishing the interface for Interactive Brokers and Oanda. Interactive Brokers seem to also allow trading in things other than just currency. And I think Oanda has the best setup in terms of the API and their spread, but their API is also the most expensive at $600 up front.

I'm guessing the normal course of events would be for someone to try out my software using a Gain Capital demo account, and then if they like it and want to use Oanda, then they pay the $600 to Oanda for the use of their API.

Opening an account with Gain Capital is quite straightforward - an online form and then a couple of things to print and fax to them if you are not a resident of the United States.

After those three are done, I will go about coding the interfaces to any other broker that provides an API.


Data
In the initial release the companies that can used for downloading price data will obviously be restricted to Gain Capital. Aside from the free API, the other good thing about Gain Capital is that they provide tick data going back many years, available for free from their website. And of course Thinking Stuff provides the ability to load the data from those files into the database.

There are one or two abnormalities in some of the files, but I will provide clean versions of those particular files.

You set what price data you want to download (which currencies, which time intervals - 5 minutes, hourly, daily, etc). And you can also specify what price data you want Thinking Stuff to create.

For example, you set that you want to download tick data from Gain Capital. You then specify that you want Thinking Stuff to create 5 minute interval data, hourly, and daily bars from the tick data. Or maybe you create the hourly data from the 5 minute, and the daily from the hourly. Up to you.

Example #2, you want to trade using 4 hour interval data, but Gain Capital doesn't provide that data (for example). So, you specify that you want to download the hourly bars from Gain Capital, and you get Thinking Stuff to create the 4 hour interval data from the hourly data that you downloaded.

Your account information and transactions (for those companies that provide transactions) can also be downloaded at regular intervals that you specify.

Access To Data
Thinking Stuff allows you to save the data to text files.

You also have *complete access* to the database. Hey, it's your data. (Just please please please don't change any of the settings that Thinking Stuff needs to run. I'll explain more about the database tables closer to/after the initial release).

Essentially Thinking Stuff runs on a client/server model, but your computer is both the client and the server.


Systems
As mentioned about 100 times or so, you create your systems by clicks of the mouse.

For both long and short, these are the settings:

• Entry Rules - all rules must be met in order for an entry to take place.


• Entry Values - if all Entry Rules are satisfied, will set the entry at this level.


• Initial Stop Loss Values - if all Entry Rules are satisfied, will set the initial stop loss at this level. Without an initial stop loss, no trades will be entered.


• Initial Take Profit Values - if all Entry Rules are satisfied, will set the initial take profit at this level.


• Move To Break Even Rules - all rules must be met in order for the stop loss to be moved to a position of break even.


• Stop Loss Management Rules - all rules must be met in order for the stop loss to be moved.


• Stop Loss Management Values - if all Stop Loss Management Rules are satisfied, will move the stop loss to this level.


• Take Profit Management Rules - all rules must be met in order for the take profit to be moved.


• Take Profit Managmenet Values - if all Take Profit Management Rules are satisfied, will move the take profit to this level.

There are also secondary Stop Loss Management Rules and Values, and secondary Take Profit Management Rules and Values.

You don't need to use all of those - in fact you can consider a system with only Entry Rules, Entry Values, Initial Stop Loss Values, and Initial Take Profit Values as being "complete".

This is where you really need to be careful and think about what you have set up. Looking at the results of the backtests should hopefully let you know if your system has all the parts it needs to properly buy and sell at the times you want it to.


Backtesting
Select the system(s) to backtest, select the currency(ies) to backtest them on, select the dates to test over. Click start.

The dates you select are for the lowest interval. Thinking Stuff works out what data it needs for the other intervals.

For example, you have a system which runs off hourly bars. But you also have a rule in there that checks a condition based on the daily bars (yes, you can create systems with multiple time-frames). You select the dates on the backtesting window that the hourly will run on, and Thinking Stuff works out how much daily data it needs.

Unfortunately the initial release will provide the results in a comma-seperated file, which opens in Excel. I say unfortunately because it's not the prettiest of beasts. The thing to remember though, is that even if I pretty it up, it's still just the same information.

The file contains all the rules that make up your system, so that if you modify the system later you will now what was actually tested.

Then the summary, then details of each trade.

You also have the option of creating a data file, which will print out the data it used for the backtesting, along with the indicator data. I have an Excel macro which makes a chart so you can see exactly where the system bought and sold and moved its stop loss, etc.


Auto-Trading
Create your system, attach an account to it. Move it from an "Available" trading system to an "Active" one. That's it.

To turn a system off, move it to the "Trade Management Only" section. When no more trades are open, it will revert to an "Available" system.


Alerts
Alerts based on if a system is placing an order, cancelling an order, entering a trade, exiting a trade. All or some or none.

Alerts based on the price reaching a certain level. Alerts based on indicators reaching a certain level. Alerts based on bar formations.


Money
I subscribe to the ideas laid out in the book called The Science of Getting Rich. I must therefore give to you more in use value, then you give me in cash value.

If you think the following rates are excessive - too expensive, or too cheap, please leave a comment. All dollar amounts are in U.S. currency.

Downloading data is free.
Backtesting is $200 per year.
Auto-trading is $30 per month.
Alerts are $30 per month.

You have complete access and control over the database in which Thinking Stuff puts the data, and as mentioned, downloading the data is free. If you can code, you might want to use Thinking Stuff to download the data, then make your own program to do the backtesting.

That's what I would do. If... I hadn't just spent more than a year creating this software. A lot of my spare time went into this. Too much of it.

You should ask yourself what your time is worth? If it's $10 per hour, you have just 20 hours in which to create something equal to what I offer. After that, you're actually losing money just because you know how to code and don't want to pay for someone else's program. Up to you. Personally I think my spare time is worth a lot more than $10 per hour.

And I have an offer for budding programmers. If you create something which utilises or is compatible with the Thinking Stuff database, I'm happy to incorporate it into my software. Either I'll buy the code from you outright, or we can agree to some large percentage of the proceeds from that particular functionality headed your way. Please use Visual Studio .NET.

More life to all.


Things that might possibly get added in the future
I'm not promising these things will get done, so you're better off assuming you never read this paragraph. However, if you really really really think something is a fantastic idea, please don't be shy in telling me so.

• Translate a trading system into TradeStations's EasyLanguage, WealthLab's WealthScript, MQL4, etc.


• Allow comparison between indicators. For example, Simple Moving Average with period 5 is above Upper Bollinger Line with period 20.


• More Equity Curve Analysis rules (research also required).


• Handle stocks, futures, etc. Not just currency.


• Yahoo! Finance as a data company.


• Allow user to select the time zone to display dates in. Currently all GMT.


• Save user settings.


• Pip-by-pip backtesting. This would allow holidays to be incorporated into backtesting, allow backtesting of scalping techniques, and remove some of the assumptions that must be made when backtesting using interval data. Would be slow though.


• Incorporate support and resistance lines into backtesting. User would specify where the lines should be placed, from when they start being used in calculations, and when they stop being effective.


• Optimisation functionality.


• "Time of day" rules - to allow entering and exiting at particular times of the day.


• Can pause/restart trading by sending an email with particular words/password to an email account.

Finally, I wanted to say that while the software is called "Thinking Stuff", it will in fact do exactly as you tell it. Nothing more. It is a useful tool, it might give you an edge over people that don't have it, but having it does not guarantee any kind of financial windfall. You still have to come up with the strategies by yourself. Or you could buy a system from somebody... Now that you have the means to test it out to see how good it actually is... :-)

Of course, the best way to tell if this is for you or not, is to trial it yourself. I'll get cracking.

Questions/comments/suggestions are most welcome.

Disk Utilities

I came across a link to some freeware disk utilities while reading some forums this morning. I haven't used them, but they look very useful.

Click here to go there, and use his FAQ page if you encounter any problems.

PWB#4: Impossible Entries & Exits

I hope everybody knows what a "gap" is.

Well, sometimes your backtesting routine will determine that it needs to set a long entry order at a particular price. And probably your backtesting routine will say "ok, our intended entry price is at this level, did the price go above that point? If so then we're in the trade". Or something like that.

And your backtesting routine will record that it entered long at that price:
21-Nov-2005 18:00:00, Entered long trade at 1.2995.

Problem is though, that just because the price went higher than your intended entry price, it doesn't necessarily mean that you were able to open the trade at that price. Because of gaps.

You can't just assume that you were filled at 1.2995 - you have to check if the open price of the bar was below your intended entry first. If it was, then you can say you entered at 1.2995. If it wasn't, that's where your last gasp entry price comes in - was the open below that? If it was, then you can say you got in at the open price. If the open was above the last gasp, you didn't enter the trade.

What if the price was higher than your last gasp, and then came lower after the open? Did we enter the trade or not? Well, I tend to think that if the open is higher than the last gasp, then we cancel the order. We don't know what's going to happen in the future. And if you enter long while the price is going down, that seems to me to be the wrong thing to do.

The same rule applies for exits, although we don't have a "last gasp" here. The rules say we want to move our stop loss to 1.2995, for a break-even setting. The next bar opens above that price, and drops through it, then we can say we exited at 1.2995. But an open below our stop loss means unfornately we just have to get out at whatever the open price was. There's no waiting for it to come back. That's praying. Maybe it will and maybe it won't. Stop losses are about risk management, and must be followed.

That's why in a system where you set the stop loss to be exactly 50 pips below the entry, you might still end up with a loss of 55 pips or something.

So that's gaps covered. There's also this thing called slippage. You might like to include a standard slippage of a number of pips just to be on the safe side. You want your backtesting results to be more negative rather than more positive. Slippage is where you want to get in at 1.2995, but your broker fills you at a higher price. Common with stocks and options. Less so with currency, but still possible.

Essentially, whatever your profit/loss was, just deduct the slippage from that and calculate a new profit/loss.

PWB#3: Getting the Bid & Ask Mixed Up

Long
Once you've set your entry price, it's the Ask price that determines if your order is taken up or not. Or if you place a Market Order, you get in at the Ask price.

Once in the trade, the Bid price determines if your Take Profit or Stop Loss is hit. Or if you Exit At Market, you exit at the Bid price.

Short
Once you've set your entry price, it's the Bid price that determines if your order is taken up or not. Or if you place a Market Order, you get in at the Bid price.

Once in the trade, the Ask price determines if your Take Profit or Stop Loss is hit. Or if you Exit At Market, you exit at the Ask price.

That's for currency anyway - with stocks and such the price is the price is the price. But don't forget the commission/brokerage (just to repeat myself).

PWB#2: Opening & Closing Trades In The Same Bar

Let's say you have only interval data (open, high, low, close) on which to perform your backtesting (i.e. no tick data).

And let's say your system wants to enter long at the high of the most recently closed bar, and it wants to set it's stop loss at the low.

And let's say the following bar has a higher high than the one used for the entry and stop loss calculations, and a lower low. There are three possibilities here:

Firstly, the price may have first risen above the high of the previous bar, thus triggering the long entry. We're in the trade. And then the price dropped below the low of the previous bar, thus trigging the stop loss. We're out of the trade at a loss.

Secondly, the price may have first dropped below the low of the previous bar, thus triggering nothing because the stop loss doesn't come into effect until we're in the trade. Then the price rose above the high of the previous bar, thus triggering the long entry. We're in the trade.

Thirdly, the price may have first dropped below the low of the previous bar, thus triggering nothing because the stop loss doesn't come into effect until we're in the trade. Then the price rose above the high of the previous bar, thus triggering the long entry. We're in the trade. And then the price dropped below the low of the previous bar, thus trigging the stop loss. We're out of the trade at a loss. While unlikely, in fact the price could have rebounded many times between the high and low of the bar during the time the bar was formed - there's no way to know if tick data is not available.

This third scenario forces us to have a policy of once being stopped out of a trade, we wait for the bar to complete before calculating a new entry point (as opposed to using the old entry price calculation to get back into a trade if the price rises to that level once more). That's my policy, anyway. Backtesting would be impossible without it as we would not be able to predict the results in this situation.

Anyway, the first and second scenarios alone highlight a problem with backtesting using interval data. Two completely seperate outcomes are possible. And so, I think there is no other option but to either (a) track both possibilities and show two sets of results at the end of the test; or (b) always assume the worst case - that the price first went in a direction in order to get us in to the trade, and then went in a direction that took out the stop loss.

Feel free to ask questions if I didn't explain that very well.

PWB#1: Forgetting About The Spread

[PWB = Problem With Backtesting]

Yes, yes, forgetting about the spread (in currency trading, or forgetting about the commission for traders of other things). Essentially, forgetting about any miscellanous costs of trading.

Simply, if you can make $5 each and every time you trade, you'd become rich very quickly. Unless... it costs you $10 in spread/commission/other each and every time you trade. And sometimes people forget that $10 in spread/commission/other bit.

Here's an example. A while ago I thought I'd stumbled onto the world's most simplest yet profitable trading system. I couldn't believe others hadn't thought of it. I was giddy with excitement.

The system I thought of, on the 5 minute chart, was to set a long entry order at the high of the last 5 bars, with stop loss at the low of the last 5 bars. Each bar move the stop loss to the low of the last 5 bars. Eventually, hopefully, the low of the last 5 bars would rise above the initial entry point, and I'd make a profit. Reverse for shorts.

You are laughing because that's a simple breakout system, and it's been around since the dawn of trading. I know that too... now.

Anyway, I was looking at the EURUSD 5 minute chart of the bid (in Oanda's charts you can chart the bid or ask or the average), and "backtested" over about 2 days worth of data. The end result was faaaantastic: 112 pips profit. 112 pips in that short time! If you trade 1 contract and make 112 pips profit, that equals US$1120. All I could see were dollar signs.

80 trades were executed in those 2 days. The problem I had with the fantastic system that I'd "created" was that I wouldn't have been able to sleep - the system required constant attention as it traded so often. (Hmmm, might need a trading-bot to help me out, I might have thought at the time :-)

And the other problem was that I'd forgotten all about the spread. The prices I worked off were all bid prices. Take the spread into account - 1.5 pips at Oanda times 80 trades = 120 pips. 112 pips profit minus 120 pips spread = -8 pips.

Excitement giddiness... fading...

The Problem With Backtesting

My next few posts will cover the common mistakes people make with backtesting.

No, wait, that should read "the mistakes I made with backtesting".

Backtesting, I think, is essential. You've probably heard over and over again that past performance does not necessarily reflect future results. And that's very true - essentially the future prices would need to be exactly the same as the ones you tested it on for you to guarantee the system's performance. Which won't happen. Probably.

But surely you wouldn't trade with real money on a system you knew to be a dog for the last couple of years?

So I guess backtesting lets you filter your list of systems to a just a couple that have been performing well up to now.

This should all tie in with your Equity Curve Analysis. You only trade real money on the systems that meet your criteria for allowing real money to be traded. If you're just starting off, then the only way to know if it meets you criteria is to backtest and found out.

Or you could paper trade every system you have thought of in real time, until one or two of them meet your critiera. But I don't think anyone has the patience for that. So it's off to backtest town for us.

But, and here's the thing, you can do backtesting the wrong way. And when that happens you're in possession of faulty results, and you'll be making your decision to trade a system on faulty information.

Here's a tip - whenever you create a system that has mind-boggling results - results which would make you rich beyond your dreams in about 6 months, and leaves you kind of giddy with excitement - you've got a bug in your backtesting process. For sure. It's happened twice to me. The good part is, after you get over the depression brought about from realising you're not going to be rich in 6 months, you've found your bug and your backtesting process has become that little bit better.

So, the next few posts will be about the mistakes that I made, and hopefully you won't need to make them yourself.

Syndication, RSS, Atom, And All That

I've decided to utilise a free service called FeedBurner to look after my site's feed. I'm not exactly a pioneer in this regard - it's fairly popular. But if you're actually reading this through your feed reader, you might like to unsubscribe to my previous Atom feed, and re-subscribe through FeedBurner. Everything you need to know below.

If you don't, this counter won't include you :-(

Now, if you're asking what any of that meant, you're not alone. And actually you really don't need to know what any of it means. I personally get by just fine without using other peoples' site feeds because I prefer to visit their websites. People format their posts to look good on their website, and it may or may not (probably not, actually) also look nice in the syndicated page.

If you do want to find out what it all means, you can start with the FeedBurner explanation. Or the Wikipedia's entry on Web Syndication. Or the search engine of your choice. I use Google. And here's what the guy at 43 Folders had to say when he did the same thing.

So, the site feed address is http://feeds.feedburner.com/thinkingstuff

You can also click:
Subscribe to Sharky's Trading Diary

Or you can click on this little picture:

Or you can click on any of these to load the feed into that particular feed reader service:
    
Enjoy!

How Markets Really Work

I've mentioned through my previous posts that you should experiment with different ideas. Unless you've purchased a system that actually works fantastically, you'll probably find yourself tinkering with it anyway. And unfortunately that kind of tinkering is along the lines of "well, if I had of used this new rule I've just made up, it would have prevented the loss I just took".

Sure, that new rule may well have prevented the loss you just took, but you have no idea how well that rule works over a longer period of time. You only know how well the rule worked for that last trade, and anything will work once. That new rule may just as well turn your slightly profitable system into a big loser over the longer term.

The moral of the story is to backtest. And backtest properly, which is the topic of a future story :-)

The other reason you should experiment, is because not all the truths you've read about the market are actually true. Or maybe they were true once, back when technical analysts drew charts by hand and there weren't many of them around (technical analysts, that is).

For example, when the price closes above a short-term high (e.g. the high of the last 5 or 10 days), a lot of books say this is a buy signal. Breakout systems are based on this principle.

Another example is when the price closes higher for two or more consecutive days in a row, a lot of books say this is also a buy signal. I think the candlestick pattern is called the "advancing three soldiers" or something like that.

But, a book I've just read, "How Markets Really Work", says the opposite. In fact, it says you're far better to be buying, or entering long, when the price closes below the short-term low. It also says that you're better off entering long when the price closes lower for two or more consecutive days in a row.

The book takes a look at a bunch of other commonly held beliefs about the market as well, and backs everything up with statistics. If you want some ideas to experiment with, and perhaps want to look at the market in a slightly different way to the crowd, I recommend you get your hands on a copy.

What must be remembered though, is that only the S&P 500 and Nasdaq 100 were looked at, positions were entered based solely on whatever was being tested (close above 5-day high, for example), and closed 1, 2, and 7 days later. As I said before, this book will give you ideas to run with, and not trading systems in their own right.

Why PostgreSQL?

I wrote a post titled "Why MySQL?" a while back explaining why I had chosen MySQL to be the database for my software (more explanation on the software here).

Essentially, I was looking for something that was as cheap as possible. MySQL is free if you use it for private purposes.

And, to store tick data, as you can download from Gain Capital's website for example, you need an industrial-strength database application. Microsoft Access won't cut it. It can only grow up to 2Gb.

While Microsoft offers a scaled-down low-cost version of its SQLServer, the very fact that it's scaled-down again means it's not big enough for my purposes.

The problem with MySQL is that if you want to make a commercial application, as mine is turning into (in a "let's run it up the flagpole" kind-of-way), suddenly a licence per machine costs about US$600. Meaning anyone who actually wanted to purchase my software would have to fork out $600 to MySQL, *and then* I could add my price on top of that.

Fortunately, someone read my original post and told me of PostgreSQL. A completely free industrial-strength database application. Reading a bit more about it found that it's a very close competitor to MySQL - superior in some ways, not in others. But completely free. And so I decided to use that.

That left me with the huge task of migrating my code to connect with PostgreSQL instead of MySQL. They are not huge changes, just a lot of them. And then I thought, hey, instead of replacing the MySQL code with PostgreSQL, why don't I *add* the PostgreSQL code. Then, people could decide which database they wanted to use.

Some people might already have paid the $600 to MySQL. If I completely swapped to PostgreSQL, I'd be forcing them to install another industrial-strength database application on a machine that already had one. Sure, computers these days are quite powerful, but you don't really want two of the things installed if you can help it.

And then I thought, hey, *I* want to store tick data but maybe some people are quite content to store daily data. In that case, Microsoft Access will work just fine. Or the cut-down version of Microsoft's SQLServer. And then if I've made the software able to connect to the cut-down version of SQLServer, I dare say it'll work just fine with SQLServer itself.

Today I came across yet another completely free database application called SQLite.

So, if you've already got any of these database applications on your machine, you'll (very hopefully because I haven't actually started making code changes yet) be able to use the Thinking Stuff software with it.

Be warned, however, that I have not checked into the specific licencing terms given by Microsoft and MySQL for when shipping commercial software that "can use" those products, as opposed to software "specifically for" those products. It could be the case that a fee is still required, albeit smaller.

A very quick recap on the software that I'm making (and have been making for oh-my-god way too long now): It's automated trading software. But, you don't need to know or learn any programming languages in order to use it (e.g. TradeStation's EasyLanguage, WealthLab's WealthScript, MetaTrader's MQL, etc). All trading systems are created by selecting rules from pull-down menus. Once you've created a system you can backtest it. And then you make it trade in your demo account. And then when you're completely satisifed, you make it trade in your real account. To be compatible with Gain Capital, Forex.com, Interactive Brokers, Oanda, plus any others that provide an API. Currently only for forex, but hoping to move to stocks, futures, as well. Hoping to release a Beta version ("Beta" meaning you can use it, but expect bugs, and tell me about the bugs when they happen) in early to mid December.

My Christmas present to myself will be to get the darn thing finished :-) And if nobody buys it, well, I made it for myself anyway.