Actionable articles must meet at least 5 out of 8 of the following criteria:
Works across business cycles
- The pattern first occurred before 2005, covering 2 expansions and 1 global recession
High signal to noise ratio
- The condition appears infrequently but distinctly, holding true a maximum of 10% of the time
Occurs frequently enough in the past
- A similar scenario occurred at least 5 prior times
Occurs at least every 5 years
- The pattern appears frequently enough to avoid long gaps, at least once every 5 years
High in-sample win ratio
- The pattern is directionally correct > 75% of the time in-sample
High out-of-sample win ratio
- The pattern is directionally correct > 75% of the time out-of-sample
High PnL Asymmetry
- Convex trade with high upside and low downside. The top 80% percentile gain is at least 1.5x the top 80% percentile loss
Hypothetical P&L is smooth with a low Drawdown Ratio
- The backtest PnL has a sortino ratio in excess of 1.5x (total return >1.5x the deepest drawdown)
Back testing - walk forward out of sample test
To test the robustness of a forward price pattern, we analyze the accuracy of its historic predictions, removing any look-ahead bias. This is accomplished by reserving a portion of the data series for analysis, and using the remainder for testing the resulting forecasts.
Using only the data history up to an individual signal episode, we predict the direction of the move and examine whether or not the forecast came true. If the ex-ante and ex-post returns match in direction, we count that as a 1, and otherwise 0. We repeat this procedure for each episode up to the current signal. The resulting ratio of accurate predictions to total episodes gives a sense of how well the signal has done historically.
Lets take an example of a TOGGLE insight which has 50 similar past occasions: to be extra sure that we aren't just finding some spurious correlation, we want to go back and test EVERY past prediction TOGGLE would have made at the 49th episode, and 48th episode, and 47th episode etc. So we go to the 49th episode and examine what return would you expect if TOGGLE was only allows to train on the data from episodes 1-48. We then compare the actual return X months later (which we know) with the return suggested by history up to that point. Then we repeat that for episode 48. And episode 47. And so on ... Each time we allow TOGGLE access to less and less data and observe the accuracy of the subsequent prediction. If the out-of-sample accuracy reaches a minimum threshold, it is awarded one of our Stars in our insight robustness ratings.