Downloading price history

Scheduled process run from the Command Centre. Some run at recurring intervals of your choosing, such as every minute, every 5 minutes, every 30 seconds, every 24 hours, or whatever. Others are "streaming" in that it's more like flicking a light switch on or off.

These scheduled processes do various things. Some download price bars from a broker, some create price bars, one kicks off automated trading, another alerting, another downloads events from the Forex Factory calendar and converts them to holidays. And so on.
 

Scheduled processes for downloading price bars

There are streaming downloads, recurring downloads, special recurring downloads, backfills, and historical prices.

Streaming downloads lets ATM receive prices from the broker as soon as the broker sends them. The brokers are always broadcasting the new prices, and by turning on the streaming process what you're telling ATM to do is log in to that broker and start saving the prices it receives. Mostly brokers offer tick prices in this way, and then you would need to also run the recurring process for that broker that creates price bars.

Recurring downloads lets you grab a few bars at regular intervals. Special recurring downloads target only those price bars that are needed for trading systems that are auto-trading right then at that time (and active alerts).

Backfill refers to trying to get price bars you might have missed, had your computer been turned off, disconnected from the internet, or crashed.

Historical prices are those going back potentially many years.

Not all brokers offer all options. But yet there are all of these scheduled processes for all the brokers. The ones that don't actually work have an explanatory note, telling you what to use instead. If having those that don't work still appear in the list is a bit strange, you can filter the list to only those that can be run.
 

This tutorial starts from a brand new settings file and goes through to downloading historical prices from FXCM.

It assumes:

1. You installed the FXCM bits when you installed ATM.
2. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
3. You used this information to go and open an FXCM demo account, and also you logged in to that account using the FXCM trading platform and got your account number.

In this tutorial:

1. Try to download prices, but realise you need to create a Data Manager entry first.
2. Try to create a Data Manager entry, but realise you need to create a User first.
3. Enter the FXCM demo user and account details.
4. Create the Data Manager entry for EUR/USD, Daily, FXCM.
5. Start the download.

This is the opposite order to that demonstrated in this video tutorial. But actually the method on this current page is more intuitive and also works out to be a bit quicker.

Anyway... let's download some EUR/USD Daily bars...

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

This tutorial starts from a brand new settings file and goes through to downloading historical prices from FXCM.

It assumes:

1. You installed the FXCM bits when you installed ATM.
2. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
3. You used this information to go and open an FXCM demo account, and also you logged in to that account using the FXCM trading platform and got your account number.

This tutorial does these things:

1. Enters the FXCM demo user and account details.
2. Creates the Data Manager entry for EUR/USD, Daily, FXCM.
3. Starts the download.

We, as creators of the software, know the pre-requisites and create them first. However, a normal person just knows that they want to download prices, so they might try that first without entering any of the pre-requisite information. There's a sister video tutorial doing it that way as well.

Ok, so... let's download some EUR/USD Daily bars...

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

ATM can only log in to one FXCM account at any one time.

If you've setup ATM to run in many FXCM accounts, then ATM will have to log out from one, then log in to another, before it can place any trades.

Logging in to the FXCM trading platform, you can notice a distinct pause after you click the "login" button, while all your account information is downloaded and displayed. ATM has to download that same information and will have the same pause while it does so.

For example, when ATM downloads the account information for all FXCM accounts you have set up, here's what happens:

1. Log in to account #1 (with the noticeable pause)
2. Log out from account 1
3. Log in to account #2 (with the noticeable pause)
4. etc.

Logging out doesn't take long. So ATM will stay logged in wherever possible, in the hope that the next action it has to perform is for the same FXCM user that it's already logged in with.

But how to get around this restriction? Is there a way to setup ATM to trade in multiple FXCM accounts on a single login? Here's what the FXCM forum says:

You can request to set up a PAMM account, which allows you to login to a manager account and trade for all accounts mapped under the manager account.

It is possible to have more than one account in the accounts window. Additional accounts would be set up through our sales and support team along with the necessary paperwork, whether they are additional accounts for you, or accounts of others' that you are managing. Contact admin@fxcm.com to start the process.

One note is that if the accounts are on different servers, they would require separate logins, even if they are both your accounts, or the account owners have assigned you power of attorney to trade their account.

(Noting that you would only be able to find out if your accounts are on different servers by contacting FXCM).

I imagine it would be almost impossible to include a demo account and a live account under the same login. It is recommended, if you want to auto-trade in both a live account and a demo account at the same time, that you install ATM twice in two different directories. Then run both 'instances' of ATM - one for the live FXCM account, and one for the demo account. You would not need to worry about doing this though, if you do not care about the lag time required when logging in/logging out.

It would be appreciated if you have gone through the steps to make this work, that you contact us and let us know the steps you took to get it all sorted.
 

When Downloading Prices

This restriction should not have a large effect if you are just downloading prices.

If a user is currently logged in to FXCM, ATM will simply use that current connection to get the streaming price downloads. If nobody is logged in at the present time, FXCM streaming prices logs in with the "best" FXCM user:

1. The first 'real' user it comes across
2. The first 'demo' user it comes across if there are no real ones.
    

If you're going to download any of the text files containing historical price data from the brokers, there is some free software that is going to make your life a whole lot easier.
 

downTHEMall!

Especially with Gain Capital, there are *a lot* of files to download. If you do one at a time you'll probably go insane. I used to do it that way until I found a better method. Using a combination of the free Firefox web browser (an Internet Explorer replacement), and a free Firefox extension called downTHEMall!, you can download a month's worth of files with a couple of clicks.

Here are the steps:

1. Download Firefox.
2. Install Firefox.
3. Run Firefox.
4. Using Firefox, browse to the downTHEMall! extension page.
5. Click on the link that says to "Install Now for Windows".
6. When that finishes, close down Firefox.
7. Restart Firefox.

Now you're ready to get the historical price data.

Using downTHEMall! also ensures that if you are disconnected for any reason, the download can pick up from where it left off (and not have to re-download the file from the beginning).
 

Text File Cleaver

Some of the files you download, especially those from Oanda, are huge. The Oanda ones contain around 20 million rows of tick price data, and ATM will probably not be able to load a big file like that in one go because of the database timeout setting on the Options window.

Instead, please use a file-splitting utility such as Text File Cleaver. Using the default options, Text File Cleaver splits the one big file into a number of smaller files, each containing 65,000 rows. ATM can easily process those files individually within the timeout period. For the Oanda file I tested with, 299 smaller files were produced.

The reason for recommending Text File Cleaver (aside from that it's free), is that it keeps each row intact. Other file splitters just split the file after a particular number of characters, which often falls in the middle of a row of price data.

Simply follow this link, scroll down to find the "Text File Cleaver" section, download and install.
 

When you create a Data Manager entry, you specify how you want to gather the prices - by downloading, creating, or none. On the Command Centre, the scheduled processes that do price downloads do so for all the Data Manager entries where you said to download them. All good so far.

But there may come a time when you want to stop downloading a particular symbol, but continue to download all the rest. You can delete the Data Manager entry completely, for sure. But that's probably overkill.

Instead, edit the Data Manager entry in question. Choose the very bottom option - "just set up...". Click OK or Apply.

In this way all of the prices you have already obtained will still be available, but none more will be downloaded. To restart downloading, just revert to the original settings.
 

To download price data from a broker, you need four things:

1. A user set up for the broker you are trying to download prices from. The password must be entered correctly.
2. At least one account set up for that user. The account number must entered correctly.
3. Data Manager entries, where you have specified to download the prices, and choose the user you created in the step above.
4. The API for your broker properly installed on your computer.

You wouldn't have been able to save the Data Manager entry without a user selected. But you could save it without an Account for that user. Just remember that in ATM, each user needs a corresponding account set up for it before the user will actually be used for anything. It's just that one user could have many accounts, so we had to split them up.