Financial Trading Systems
By Paul H. Lasky
Traders who utilize optimized
systems and the designers of these systems face a fundamental dilemma. How to
validate the system once the system parameters have been optimized.
"Validation" means the process of assuring
that the system performance continues to be optimal or, at a minimum, adequate
when tested on future data not utilized
to determine the optimal parameters. In other words validation assures that the
optimized system was not "curve-fit".
Because all financial market data
reflects contemporary economic events, the data is non-stationary, and the system designer must select a developmental
data set that reflects current conditions. This means that the pertinent data set is limited. When the
"out-of-sample" validation method is utilized, this precious limited data must be split into separate
developmental and test sets. As a result, usually neither set is large enough
to accomplish the twin goals of adequately determining an optimal parameter set
and validating the system.
With limited test period data, the
"out-of-sample" method lacks an objective criterion for determining whether the system performance is adequate
when the variance of the test results is large. Standard deviations of 2 to 7 times the test mean
performance are typical for commercial trading systems. Clearly more "out-of-sample" data is required to
validate the system. But splitting the limited available data by increasing the test data set at the expense
of the developmental set only makes the optimization less reliable. Clearly a better method that makes more
efficient use of the available data is required.
The Validation Principle
An alternative to the
"out-of-sample" method can be developed by comparing the optimal parameter
set performance with the performance
utilizing other possible parameter sets on the same data set. Two key points should guide the search for a more
efficient validation principle. The first point is that the comparison should utilize all available data including
the developmental data set. The developmental data can be used to test the differences between optimized
systems assuming that the optimization process has a roughly equal effect on the systems being compared.
Category: Methods of technical analysis
| 
