Tasks/Scheduled processes

Scheduled process run from the Command Centre. Some run at recurring intervals of your choosing, such as every minute, every 5 minutes, every 30 seconds, every 24 hours, or whatever. Others are "streaming" in that it's more like flicking a light switch on or off.

These scheduled processes do various things. Some download price bars from a broker, some create price bars, one kicks off automated trading, another alerting, another downloads events from the Forex Factory calendar and converts them to holidays. And so on.
 

Scheduled processes for downloading price bars

There are streaming downloads, recurring downloads, special recurring downloads, backfills, and historical prices.

Streaming downloads lets ATM receive prices from the broker as soon as the broker sends them. The brokers are always broadcasting the new prices, and by turning on the streaming process what you're telling ATM to do is log in to that broker and start saving the prices it receives. Mostly brokers offer tick prices in this way, and then you would need to also run the recurring process for that broker that creates price bars.

Recurring downloads lets you grab a few bars at regular intervals. Special recurring downloads target only those price bars that are needed for trading systems that are auto-trading right then at that time (and active alerts).

Backfill refers to trying to get price bars you might have missed, had your computer been turned off, disconnected from the internet, or crashed.

Historical prices are those going back potentially many years.

Not all brokers offer all options. But yet there are all of these scheduled processes for all the brokers. The ones that don't actually work have an explanatory note, telling you what to use instead. If having those that don't work still appear in the list is a bit strange, you can filter the list to only those that can be run.
 

This tutorial demonstrates how to create a trading system, and then backtest that system using historical price bars.

It assumes:

1. You started ATM and opened a settings file.
2. That settings file already has some price bars in it.
3. ATM is sitting there ready to go.

This tutorial does these things:

1. Creates a trading system.
2. Creates a backtest configuration entry.
3. Runs the backtest.
4. Views the chart. We decide how the system might be made better.
5. Edits the trading system.
6. Runs the backtest again.
7. Views the chart again to see if we did actually make it better or not.

There are several objective ways of determining if a price is trending or ranging, and the trading system we create in this tutorial was inspired by the Two Bollinger Bands method. For a stop loss we use the Chandelier Stop described in more detail here.

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Downloading historical prices from FXCM
• Video: Loading text files and creating price bars

This tutorial starts from a brand new settings file, loads a text file of historical tick prices, creates price bars from those ticks, and charts the price bars. The file in question was obtained from Gain Capital, but the theory is the same for text files from any source.

Note, however, that the file we used in this particular case was an older one. Gain Capital's newer files are slightly different. See the settings to use on the File Loader window.

This tutorial assumes:

1. You started ATM, created a new settings file, opened it, and ATM is sitting there ready to go.
2. You downloaded some text files containing historical prices.

This tutorial does these things:

1. Loads the historical tick prices from the files.
2. Creates 10-Minute bars from the ticks.
3. Creates Hourly bars from the 10-Minute.
4. Charts the 10-Minute bars.

A very similar process is described (in words) in this post.

Click here to watch the video, and use the browser's back button to come back. Or click here to open in a new window.
 

See also:

• Video: Backtesting

The files are available at http://freeserv.dukascopy.com/exp/. Also www.dukascopy.com/swiss/english/data_feed/csv_data_export/.

But you can't start the downloading of all the files and go and do other things - you have to export the data you want one-by-one. Scroll about 4/5 of the way down to get to the currencies available. They have |f| next to them. You can choose any format you like, but make sure you choose "Windows" in the "End line" field.

The good part is that they have data going back to 1986 for some currencies.

DukasCopy provides indicative data, which has no spread. During backtesting, make sure you add on a few pips in the "Commission" field to simulate the spread, otherwise your backtesting results will be overly optimistic.

The Currency Secrets website has covered the issue of indicative data in some depth, so I point you to their pages:

• Beware Of indicative data
• Is indicative data really *that* bad?

p.s. I think their conclusion is that as long as you add the spread, it should be ok.

I guess it's always better to use the prices from the broker you are going to use, if possible.

Once they are finished downloading, follow the rest of the instructions to load text files. DukasCopy historical prices are already in price bar format though. So you don't need to do the steps to create price bars, unless you want to create other price bars not provided by DukasCopy.
 

The method described below works the same for any symbol/interval/broker. We will use the example of creating EUR/USD Hourly and Daily price bars from historical tick prices provided by Gain Capital.

These steps assume you have already downloaded the text files containing the historical prices to your computer. See here for instructions for:

• Gain Capital
• Oanda
• DukasCopy
   

Create the Data Manager entries

Here's a good explanation of what Data Manager entries do.
 

1. Create a Data Manager entry to store the tick prices

1a. In ATM, click "Price Data" > "Price data configuration"
1b. Click "Add"
1c. In the top box, choose EUR/USD, Tick, Gain Capital
1d. In the bottom box, choose the bottom option - "Just set up (or keep)..."
1e. Click OK.
 

2. Create a Data Manager entry to store the Hourly bars

2a. (Should still have the Data Manager Search window open).  Click "Add"
2b. In the top box, choose EUR/USD, Hourly, Gain Capital
2c. In the bottom box, choose the second option - "...shorter time-period..."
2d. Choose "EUR/USD, Tick, Gain Capital" from the list.
2e. Click OK.
 

3. Create a Data Manager entry to store the Daily bars

3a. (Should still have the Data Manager Search window open).  Click "Add"
3b. In the top box, choose EUR/USD, Daily, Gain Capital
3c. In the bottom box, choose the second option - "...shorter time-period..."
3d. Now you have a choice - you can either choose "EUR/USD, Tick, Gain Capital" from the list, just like you did with the Hourly bars. That would mean the Daily bars are also made from the tick prices. Or, you can choose "EUR/USD, Hourly, Gain Capital", to have the Daily bars made from the Hourly. It's up to your preference, but the second option would be far quicker.
3e. Click OK.
 

Side note #1:

Each Data Manager entry creates a file on your computer which will store the prices. If you look in your file system you should now see 3 new files: gain_32_tk.tsphd, gain_32_1h.tsphd, and gain_32_1d.tsphd, respectively.
 

Side note #2:

The number 32 is what we call the "FIP ID", or "Financial Product ID". It's a number only useful within the Thinking Stuff world. 32 equates to the currency with symbol "EUR/USD".
 
Now to get the price history from the text files into the ATM files. To do this, we first create a "file load task", then process the task.
 

Load the files

 
4. Create the file load task

4a. Click "Price Data" > "Price history file loader"
4b. Files to load - browse for and select the file(s) to load. Recommend to just choose 1 file if this is your first time.
4c. Fill in the "File format" and "Dates in the file(s)" fields as per these standard settings for your broker.
4d. Price data to be loaded - EUR/USD, Tick, Gain Capital
4e. Click "Add To List"
 

5. Process the task

5a. Still on the File Loader window, go to "The Task List" tab
5b. Click "Start Loading"
5c. Wait.
 

6. Confirm the task worked

6a. Go to the "Completed List" tab.
6b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
6c. Back to the Data Manager Search window.
6d. Right-click on the "EUR/USD, Tick, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
6e. The message that shows up should display a count greater than zero.
 
So now we have the tick prices. What comes next is creating the Hourly and Daily price bars. This time we create "interval creation tasks", then process the tasks.
 

Create the price bars

 
7. Create the interval creation tasks

7a. (Should still have the Data Manager Search window open). Click (normal left-click) on the "EUR/USD, Hourly, Gain Capital" entry to highlight it.
7b. At the bottom of that window, go to the "Bulk Price Bar Creation" tab.
7c. Click "Add To List"
7d. Click on "EUR/USD, Daily, Gain Capital" to highlight it.
7e. Click "Add To List"
 

8. Process the tasks

8a. Still on the Data Manager Search window, go to "The Task List" tab
8b. Click "Start Creating Price Bars"
8c. Wait.
 

9. Confirm the tasks worked

9a. Go to the "Completed List" tab.
9b. Scroll all the way to the right to see the result - "Success" is good. Anything in the "Error Text" field is bad.
9c. Back to the "Price Data Configurations" tab.
9d. Right-click on the "EUR/USD, Hourly, Gain Capital" entry. A little pop-up menu appears. Choose "Table Information (row count, etc)"
9e. The message that shows up should display a count greater than zero. Same for the Daily one.
 

Cleanse the prices

Not all text files are created equal. Some have minor errors which can throw out your backtesting results. The easiest way to check for these errors is to view the chart. Ridiculous prices are immediately obvious.
 

10. View the chart

10a. Click "Price Data" > "View charts"
10b. Fill in the search criteria.
10c. Click "Chart"
 

11. Cleanse the data

11a. If the chart looks fine then you're all done.
11b. If there are extreme spikes in the chart then it's possible there is some erroneous data in the text files. Follow these steps to perform the cleanse.
 

On the Command Centre, the scheduled processes that do account info downloads and transaction downloads do so for all accounts. That is, as long as the user that owns the account has a valid password entered.

There may come a time when you want to stop downloading a particular account's details, but continue to download all the rest. You probably don't want to delete the account just to achieve that.

You can kind of achieve it by removing the user's password. That will stop that account's details being downloaded. However it will also stop all of that user's accounts from being downloaded, and also any Data Manager entries associated with the user. So it's kind of possible, but maybe not worth it.
 

When you create a Data Manager entry, you specify how you want to gather the prices - by downloading, creating, or none. On the Command Centre, the scheduled processes that do price downloads do so for all the Data Manager entries where you said to download them. All good so far.

But there may come a time when you want to stop downloading a particular symbol, but continue to download all the rest. You can delete the Data Manager entry completely, for sure. But that's probably overkill.

Instead, edit the Data Manager entry in question. Choose the very bottom option - "just set up...". Click OK or Apply.

In this way all of the prices you have already obtained will still be available, but none more will be downloaded. To restart downloading, just revert to the original settings.
 

Gain Capital files are available to all - no account required.

1. Download the files from http://ratedata.gaincapital.com/ to your computer.
2. They are zip files, so unzip them.

Then follow the rest of the instructions to load text files and create price bars.
 

An easier way to download

Gain Capital gives one file per week per currency. About 20 currencies equals about 80 files per month and about 960 files per year. That's a lot of clicks. downTHEMAll! will make the downloading much easier than doing it one-by-one.

1. Browse to http://ratedata.gaincapital.com using Firefox.
2. Click through to the month of data that you want to download.
3. Click "Tools" > "downTHEMAll!" > "downTHEMall!..." from the main menu (in Firefox).
4. On the downTHEMall! window:
   1. In the Filters area at the bottom, check the "Archives (zip, rar...)" checkbox.
   2. Click the Browse button to select the directory you want to download the files to.
   3. Click the "Start Downloads!" button.