Custom Search

The Illusion of System Testing

There is perhaps, no other area in the field of futures trading which has been as abused or as misrepresented as trading system development. Futures traders generally pride themselves on being a group of objecВэtive, analytical, and logical people. They gloat in their skepticism and dogged insistence upon scientific validation of trading methodologies. Wherever possible, they insist upon the hard facts of system perforВэmance. This orientation to the validation of futures trading systems and methods has been growing in intensity and popularity since the advent of affordable computer hardware and software.

A number of authors, of whom the most prominent is Perry Kaufmann, have written extensively on the subject of system testing and system development. Great emphasis has been placed upon the importance of adequately testing trading systems in order to validate their use in real time. Furthermore, the area of system optimization has received considВэerable attention as well. Both areas should be studied carefully by serious futures traders, since the issues involved are likely to significantly impact the systems and methods you trade as well as your market performance. Perhaps my input will assist you. Possibly you may not agree with everyВэthing I'm saying. But my words come from the voice of experienceтАФ experience which spans nearly one-quarter of a century in the futures markets. First, let's deal with the issue of system development.

System Development

With a few thousand dollars' worth of computer software and hardware, it is now possible for virtually any semi-computer-literate individual to develop seemingly profitable trading systems. I use the words "seemingВэly profitable" because, in my experience, the development of trading sysВэtems which have shown profitable back-tested results does not in any way, shape, or form guarantee that these results will be reproducible in the future under real-time market conditions. Please note a system test is merely the analysis of a particular model which is imposed upon the market. The more complex the model, the more it will reflect a fit of the past data, but this fit is not necessarily applicable to future market conditions.

Curve fitting, as the procedure of developing seemingly effective tradВэing systems is called, can produce spectacular results which are nothing but hypothetical and which have no bearing whatsoever on the future. Hence, system testing deserves attention and close scrutiny if test results are to be taken seriously and used for the purpose of actual trading.

Several important issues must be understood if you are to avoid the trap of trading based on system tests. I realize that my point of view regarding the issues of systems, system testing, and system optimizaВэtion will not be shared by all readers or for that matter by all traders or market analysts. I feel it is necessary, nevertheless, for me to express these ideas so that you will, hopefully, achieve a better understanding of my orientation to the markets, and to trading futures. You will find that my point of view regarding the subject of system development is not nearly as rigid or institutionalized as is the current thinking of my colВэleagues. Although I am not opposed to system testing, I do believe that there are inherent dangers in taking the results of system tests too literally. Let me explain what I mean.