Previous Next Table of Contents

5. Getting the right answer: correctness issues

A key concern in TREE's design has been ensuring accuracy of simulation; that is, ensuring that historical testing accurately models the results that would be obtained in real time. The very structure of the system is chosen in part to assure correctness of testing.

5.1 Not looking into the future

In most software simulations of trading systems that TREE's authors have seen, it is depressingly easy to make the fundamental error of ``looking into the future''. In real life one cannot know today's closing price before market close time comes around (making money would be pretty trivial if one could). But in historical testing, that closing price is right there in the database, often just one array location away from data that one does know. It is absolutely critical that the trading algorithm under test not make use of that data before it is supposed to. Of course no one would deliberately do that, but typographical errors are easy to make and hard to spot. For example, a complex equation involving yesterday's prices and today's opening price might be used to decide what to do today. If a programming error allowed today's close to get into the equation, the results of historical testing might be better than they have any right to be --- and better than they will be when the programmer attempts to apply the equation in real time. More subtle conceptual errors can lead to the same result, and can be even harder to find. TREE's authors have seen other people make this kind of mistake many times (and made it themselves more than once). As a result of this hard experience, TREE is designed to prevent this class of error directly.

In the first place, TREE is built so that as far as possible, the same code executes for both historical testing and live trading. This eliminates simple errors of transcription. From the point of view of TS blocks, there is no difference between historical testing and live trading; they live in a simulated-time world, and generally pay no attention to whether simulated time matches real time. In live trading, we feed new price data (ticks) into the system as it arrives; after each tick we execute blocks until no events remain outstanding, then wait for another tick. For historical testing the price data comes from a stored database, and we just execute as fast as we can --- after each round of execution, the simulated time can be advanced immediately to the time of the next stored tick. The rate of clock advance and the source of tick data are concerns only of the TREE scheduler, which is outside the TS block world.

In the second place, few types of TREE blocks need direct access to the historical database; BarGen and TradeExec are the only standard ones. These blocks are the only ones on which correctness of historical testing depends. To be certain that historical testing accurately models what would happen in real time, we need convince ourselves of only three things:

  1. BarGen must not release historical data into the TS world sooner than it should.
  2. TradeExec must faithfully reflect what would happen to a trade order issued at the indicated simulated time.
  3. The TREE scheduler, which controls the execution of blocks, must not allow simulated time to run backwards.
These critical block types are deliberately kept as simple as possible, and free of ``policy'' decisions that a user might need to alter. Thus we can debug them once and forget them. All the rest of a trading system might contain errors, but those errors will not allow it to look into the future. (To put it another way: what is the difference between an error and a brilliant new trading technique? None, if it trades well. The only unforgivable error is reporting good results from ``historical testing'' of something that won't work in live trading.)

5.2 Accuracy of trade modeling

We said that TradeExec must faithfully reflect what would happen to a simulated trade order. That is not such a simple matter, and TREE goes to some lengths to make the results as accurate as possible --- and to err in the conservative direction if it must err.

TradeExec can work from tick, intraday-bar, or daily-bar data, depending on what's available in the historical database being used by TREE. For volatile markets, there is no substitute for tracking prices tick-by-tick: a drastic swing can happen in a few minutes in the S&P, for example. If tick data is available, TradeExec figures the time window within which the order is expected to be filled, and takes the least favorable price seen within that window. (Some additional rules, such as closing the window as soon as a pullback in the favorable direction occurs, improve the accuracy even further.) If only bar data is available, TradeExec estimates the trade price using approximate methods, such as a stop order's price plus or minus a slippage parameter. These methods are often good enough for modeling slow-moving markets and trading methods that do not react to intraday price changes. But things go better with ticks... so a tick-level database is TREE's preferred milieu.

Isolating these modeling concerns within a single block type improves the reliability of TREE system testing, since the necessary work to make a trustworthy trade modeler need be done only once.

Previous Next Table of Contents

Copyright 1997, Structured Software Systems, Inc.