The Six Rules of System Development, 12/07/05
A trading system developer is a modeler. A trading system is a forecasting model. Here are six rules that improve the modeler, the models and the modeling process:
1) Validate statistically
2) Validate ebb and flow
3) Validate with optimal f
4) Anticipate evolution
5) Know when finished
6) Diversify methods
Validate the Model Statistically
Using z scores, regression or other statistical methods, validate the models edge. Be wary of adding parameters to a model to improve a model's statistics because the more curve fit a model is the shorter its life span will be. Validating statistics are the first step toward believing in the outputs of the model.
Validate the Ebb and Flow of the Model
Every model will have good periods and bad periods. I use Control Charts extensively for this purpose. Control Charts give me a historical overview of the models P&L performance and can be used to turn a system off if it begins performing far outside of historical norms. Validating the ebb and flow of the model is the second step toward believing in the outputs of the model and the most crucial step toward believing in the model when its last output was wrong. (See Small Edge)
Validate the Model Again with Optimal f
Go back and validate the model a third time using optimal f. This validation provides a third check on the tradability of the model; it returns the maximum amount of capital and hence, leverage, that can be applied to the model which will tell you whether the models risks and returns are acceptable.(Note: After the Optimal f validation, the model will have been checked from three different perspectives: the core statistics, the ebb and flow of the P&L, and money management/tradability - a three dimensional set of checks and balances.)
Anticipate the Evolution of the Model
Just as every model will have good periods and bad periods, every model will grow over time. Trading experience and new entry and exit methodologies will serve as the basis for stepwise improvement. Build the model with improvement in mind, code gracefully, document will and use simple version numbering to keep model development and deployment in sync. (See A Recursive Framework for Modular Trading System Development and Building a Recursive Team to Implement a Recursive Framework)
Know When the Model is Finished
Some models lose money. Some models break. No model is perfect. All models, however, reach a point of diminishing returns - the point where the time put into them no longer justifies the return received and new opportunities delayed. Learning when a model is good enough (or bad enough) to be called done and moving on is an ROI-enhancing skill. (See It is the Adventurers Who Accomplish Great Things)
Diversify Modeling Methods
There's little advantage to finding the nth variation on a theme. Learning, applying and experimenting with new forecasting and modeling methodologies leads to stepwise improvement and to better, more innovative models over time. Diversification feeds back into model evolution and creates a growth and feedback mechanism for models, modeler and modeling process.
In summary then, a forecast:
Six rules + Three validations = Better * (Modeler + Models + Modeling Process)