Simple Tools for Testing Trading Ideas, 07/18/09
Three tools and four steps toward historical insight into tomorrow -
Three simple tools:
- Historical Data
- Software to test the idea with
- Two statistics to evaluate the test results
One simple process:
- Translate an idea into the testing software
- Run the software and generate the results from testing the idea
- Evaluate the results using two statistics: t score and optimal f
- Evaluate the evaluation (what are the results saying?)
An Example, Tomorrow, July 20, 2009:
Using TradeStation for the data and the testing platform and testing 'tomorrow' - 7/20/2009.There are two striking features about tomorrow in the S&P 500 which provide two simple ideas to test:
- the market has been up six days in a row, and
- it's up more than 5% in those 6 days.
Idea #1: What happens after c > o for each of the last five days?
In TradeStation that idea looks like:countif(c[0] of data1 > o[0] of data1, 5) = 5 // close is greater than open on each of the last 5 daysThe test is:
Buy on the close and exit one day later.The results of the test (S&P futures since 1997) are:
(The test could just as easily be an intraday or multiday test but 'what is likely to happen tomorrow' is an interesting place to start.)
44/88, 50%, avg -1.2 pts, standard deviation 8.7
(Historically there have been 88 trades matching the test, 44 have closed up the next day with an average loss of -1.2 pts for all trades with a standard deviation of the trade results of 8.7 pts which means that roughly 65% of the trades fall between -9.9 pts and +7.5 pts)
Calculate the t score and optimal f and use those statistics to evaluate the above results:
t = sqrt(observations) * average trade / standard deviation
The t score gives the odds, in standard deviations, the results of the test occurred by chance alone. Generally, results greater than 1.6 or less than -1.6 are far enough from chance that the idea is worth working with more. Think of the classic bell curve, 0 is pure chance and moving away from 0 in either direction is moving away pure chance.
t = sqrt(88) * (-1.2) / 8.7
t = -1.34
Evaluation: The expectation is negative and the results are 1.34 standard deviations away from pure chance. Not quite -1.6 but worth a bit more effort. Because the t score is a bit low, skip the optimal f calculation and refine the idea.
Idea #1b: What happens after c > o for each of the last five days and the market is up at least 1%?
In TradeStation that idea looks like:countif(c[0] of data1 > o[0] of data1, 5) = 5 // close is greater than the open on each of the last 5 daysThe test is now:
c[0] of data1 > c[6] of data1 * 1.01 // close is greater than the close six days ago by at least 1%
Sell on the close and exit one day later (because of the negative expectation)The results of this test are:
36/69, 52%, avg 11.8 pts, sd 8.3, win loss ratio 1.68, largest losing trade -$2157 (emini)Given the significant t score, calculate the optimal f statistic. The optimal f value, when divided into the largest losing trade for the idea gives the maximum leverage that can be applied to the idea and still avoid risk of ruin. Maximum leverage converts into the minimum account size required to trade n contracts of this idea.
t = sqrt(69) * (1.8) / 8.3
t = 1.8 (note that switching from buying to selling changed the sign of the t score)
optimal f = (((1 + win loss ratio) * probability of winning trade) - 1)/ (win loss ratio)
maximum leverage = largest losing trade / optimal f
optimal f = (((1 + 1.68) * .52) - 1) / 1.68
optimal f = (2.68 * .52) - 1) / 1.68
optimal f = (1.39 - 1) / 1.68
optimal f = .23
maximum leverage = 2157 / .23
maximum leverage = 1 contract for every $9380 in the account
Evaluation: The expectation is negative and the t results are significant - more significant because the results of this test are almost 2 standard deviations away from pure chance. Red flag: Curve Fitting. The results were improved by pulling specific results out of a more general test. Was it reasonable to attach a magnitude to the test? Probably because it certainly seems that 5 up days in a row that are up 'a lot' are more likely to retrace than a if the same 5 days were only up 'a little'. Curve fitting is where science meets art: The rule of thumb is general is better than specific.
Evalutation: Maximum leverage is $9380 in the account for every contract traded to avoid risk of ruin. To trade 5 contracts, the _minimum_ account size would need to be $46,902.
Idea #1c: What happens after n days? The idea here is to scan the historical results and see what happened after 2, 3, 4..n days. Sometimes this test produces some ideas for further tests.
The idea looks like:countif(c[0] of data1 > o[0] of data1, 5) = 5 // close is greater than the open on each of the last 5 daysThe tests are now:
c[0] of data1 > c[6] of data1 * 1.01 // close is greater than the close six days ago by at least 1%
Sell on the close and exit 1, 2, 3, n days later.An interesting result of the test is:
The expectation peaks after 2 or 3 days.
Idea #1d: Given the results of the multiday test above, what happens if the trade is exited on the first profitable close after 3 days?
The idea looks like:countif(c[0] of data1 > o[0] of data1, 5) = 5 // close is greater than the open on each of the last 5 days c[0] of data1 > c[6] of data1 * 1.01 // close is greater than the close six days ago by at least 1%The tests is now:
Sell on the close and exit on the first profitable close _or_ 3 days later, whichever comes first.The results of the test are:
t = sqrt(66) * 2.3 / 11.6
t = 1.6 (significant)
optimal f = (((1 + .41) * .82) - 1) / .41
optimal f = .38
maximum leverage = $2205/.38 = $5802 (a minimum of $5802 in the account for every contract traded)
Evaluation: t score is significant at 1.6 standard deviations away from the results occurring by pure chance, minimum account balance to trade one contract is $5802. The largest losing trade was over 40 points, so the ability to hold through the adverse excursion is in question. There's a bit more curve fitting but, again, it feels reasonable to give a trade like this a few days room. Exiting on the first profitable close has increased the odds of a winning trade (to over 81%) but has also decreased the win loss ratio to .4:1 so the trade now loses $2.50 for every $1 it makes over time. (And over time is _the_ key evaluation insight because these results accrue over time, they may or may not accrue on the next trade.)
Idea #1e: What happens when the magnitude parameter is strengthened to up 4% in the last five days?
The idea looks like:countif(c[0] of data1 > o[0] of data1, 5) = 5 // close is greater than the open on each of the last 5 daysThe tests is now:
c[0] of data1 > c[6] of data1 * 1.04 // close is greater than the close six days ago by at least 4%
Sell on the close and exit on the first profitable close _or_ 3 days later, whichever comes first.The results of the test are:
8/8, 100%, avg -9.0 pts, sd 10.8Evaluation: Curve fitting at its finest, the markets are not deterministic.
Conclusion
Using simple tools and a simple process its straightforward to gain insight into the odds of what will happen tomorrow. The upside of the process is a better understanding of the probabilities of tomorrow. The downside, as in the last example, is getting carried away with the process and thinking a test, even a perfect test, means that something _will_ happen.Brett's addendum and his paper on how to test trading ideas in Excel is here...