Custom Search

System Testing and Optimization

Friend or Foe?

The verdict of the world is final.

st. augustine

The days of untested systems are gone forever. In fact, the pendulum is now swinging in the other direction. While unscrupulous operators once sold systems and methods for which they claimed fantastic results, today's unethical operators use statistics as a tool of deception. These individuals who, paradoxically, will benefit from the trend toward the statistical validation of systems can easily dupe the public. Manipulating statistics is not difficult. Just as Archimedes once said, "Give me a place to stand on and I can move the earth," the modern systems promoter would likely say, "Give me enough statistics and I can prove anything."

This sermonette on system validation makes the point that merely testing a system and generating highly favorable hypothetical results does not guarantee success with that system. Nor should such statistics be used as a security blanket or crutch by traders. Statistics can easily be manipulated, systems can be (and are) curve-fitted, and results, unless realistic, will not reflect actual performance when the system is implemented.

While many systems are developed to show optimum performance, it is imperative that systems be tested to show the worst-case performance.

Why Test Trading Systems?

Traders test systems for various reasons. Some test a system merely to say they've done so, only to disregard the outcome or to accept mediocre results, rationalizing the negative aspects of their system. Other traders test systems in order to sell them to the public - their goal is to optimize systems in order to show maximum performance. Then there's the serious futures trader who tests systems to achieve several goals, including but not limited to the following:

ж To determine whether a theory or hypothetical construct is valid in historical testing

ж To summarize the overall hypothetical performance of a system and to analyze its various aspects in order to isolate its strong and weak points

ж To determine how different timing indicators interact with one another to produce an effective trading system

ж To explore the interaction of risk and reward variables (i.e., stop loss, trailing stop loss, position size, etc.) that would have returned the best overall performance with the smallest drawdown

Test Your Trading System

While it may seem that the last item listed above refers to optimization, you will see from the discussion of optimization later in this chapter that it is not optimization according to my definition of the term. The purpose of testing systems is simply to find what will work best for you based on what appears to have worked best in the past. In so doing, we must remember that what worked in the past in hypothetical testing may not necessarily work in the future.

A thorough test of your trading system should include at least the following information:

Number of Years Analyzed. Although it is desirable to test as much data as possible, many trading systems and indicators do not withstand the test of time. The further back you test, the less effective most systems will be. Many system developers test only 10 years of historical data, since that best shows their systems. You must make your own decision regarding the length of your test.

Number of Trades Analyzed. More important than the number of years analyzed is the number of trades. You need not analyze many years of data if you have a large sample size of trades. I recommend at least 100 trades, provided your system will generate this number of trades in back-testing. If you are truly interested in determining the effectiveness of your system, the more trades you test, the better. Remember that there will always be a tendency to test fewer trades when you realize that the system is not holding up under back-testing. Some traders argue that the factors underlying futures market trends 25 years ago were distinctly different from those during the past 10 years. They feel that testing 25 years of data distorts the picture. If they were correct, how would we know when the current market forces change and that we must therefore change our trading systems? We are much better off finding systems that work in all types of markets.