Turkey or Trader?!Jeremiah Lowin, CFA!

QuantCon — 14 March 2015!

With great thanks to Quantopian!!

Quick Survey!

•  FI: Trader / Risk / Investment Professional!

•  QU: Statistician / Data Scientist!

•  CS: Computer Scientist / Developer / Programmer!

•  NA: Non-professional / Hobbyist / Amateur!

I am a professional skeptic.!(And so can you!)!

Turkey or Trader?!

Turkeys !…make naive forecasts about the future!…accept circumstances without question!…focus on results!!

Traders !…base decisions on evidence!…are skeptical about their circumstances!…focus on generative processes AND results!

Statistics 101!

•  Statistics is the science of uncertainty!

•  Null Hypothesis: falsifiable statement about the world!

•  Hypothesis Test: determines whether we reject the null in favor of an alternative hypothesis !

•  Type I Error: when we wrongly reject the null !(a false positive)!

Type I Error!

Source: XKCD #882, “Significant”!

(Turkeys)

A “Data Science” View of Investing!

•  Markets are extremely noisy (and possibly random) machines that efficiently turn opinions into numbers!

•  Null hypothesis: market behavior can not be consistently or sustainably predicted!

•  How to test this hypothesis?!

Backtest!!

Backtesting!!

Expectation: !!

Reject the null hypothesis by discovering a lucrative strategy through rigorous quantitative research!!!!

!Reality:!

Why Type I Errors?!1.  Incentives: type I errors are fun!!

•  Type I error + great marketing = billions of dollars!

2.  No strong framework for evaluating backtest quality!

3.  Biased toward making errors!

•  We rarely view good backtests as “bad” but often decide bad backtests are “good”!

•  We can get good backtests without predicting the market (e.g. by cheating)!

Backtest as Hypothesis Test?!•  Highly multidimensional statistic!

•  No sufficient statistic or goodness-of-fit metric!

•  Biased estimator!

•  No standard error for backtest-derived metrics !

•  Therefore, it may not be easy or possible to compare backtests to the null or even each other!

Pseudo-

Backtest as Hypothesis Test?!•  Qualitative backtest evaluation framework:!

•  No knowledge of algorithm (to start)!

•  Realistic (backward-looking)!

•  Representative (forward-looking)!

•  Quantitative evaluation is not always possible!

Pseudo-

Realism!You are what you backtest!

Backtest vs Statistic!

•  Backtest: measures if a relationship can be traded!

•  Statistic: measures if a relationship exists !

•  Don’t get them confused!!

Predict SPY from Proprietary Signal!

-------------------------Summary of Regression Analysis------------------------- Formula: Y ~ <x0> + <x1> Number of Observations: 922 Number of Degrees of Freedom: 2 R-squared: 0.8359 Adj R-squared: 0.8356 Rmse: 63.1617 F-stat (2, 920): 2343.6638, p-value: 0.0000 Degrees of Freedom: model 2, resid 920 -----------------------Summary of Estimated Coefficients------------------------ Variable Coef Std Err t-stat p-value CI 2.5% CI 97.5% -------------------------------------------------------------------------------- x0 -0.0430 0.0197 -2.18 0.0292 -0.0816 -0.0044 x1 5.2362 0.1852 28.28 0.0000 4.8732 5.5991 ---------------------------------End of Summary---------------------------------

Predict SPY from Proprietary Signal!

Looks like a backtest, quacks like a backtest!...isn’t a backtest.!

That Twitter Paper!

•  Bollen et al, 2010, “Twitter mood predicts the stock market”!

•  Claims 87.6% prediction accuracy [or was that 86.7%?]!

•  Interesting normalization step:!

Source: http://arxiv.org/pdf/1010.3003v1.pdf!

Replication!

Source: http://cs229.stanford.edu/proj2011/GoelMittal-StockMarketPredictionUsingTwitterSentimentAnalysis.pdf!

Look familiar?!!

Best predictor of tomorrow: today!

(on average)!!

Not a backtest!!

Backtest Engines!

•  Backtest engines: laboratories for quants !

•  A realistic backtest engine does not guarantee a realistic backtest!!

•  (…but it sure helps. Thanks, Quantopian! )!

Lookahead Bias!

More Biases!•  Survivorship: include/exclude securities based on current

knowledge!

•  Hindsight/Loss aversion: ignore drawdowns because the strategy recovers later!

•  Hindsight/Exclusion: don’t test strategies that we “know” won’t work!

•  Recency: discount historical returns in favor of recent ones!

•  Confirmation: seek to accept hypotheses, not to reject them!

•  Confirmation/Overconfidence: reluctance to abandon an idea (IKEA effect)!

Representation!How to succeed at backtesting without really trying!

Outliers Really Matter!

•  Seems obvious, but often ignored!!

•  NOT because of black swans (but that’s important too!)!

•  Quant Rule #1: the future will behave like the past!

•  Outliers, by definition, don’t look like the past!

Outliers Really Matter!

171σ!

Story-driven Investing!

•  The opposite of a black swan:!

•  Extrapolating from outliers, or a chain of highly specific and unlikely events!

•  Certainly not representative, possibly unrealistic as well!!

Skip 3 Worst Days of the Year!(bonus points: neither realistic nor representative!)!

Leverage Really Matters!

•  Levered strategy + unlevered benchmark = win! !

•  Levered strategy + finite capital = game over!

2X SPY (Apples & Oranges?)!

10x SPY (3/2009-)!(“Hi, I’m new to Quantopian!”)!

10x SPY (1/2009-)!(“Hi, I lost all my money!”)!

(subject to margin considerations)!

Path Dependence Really Matters!

•  Backtest curves are highly path-dependent and easily distorted!

•  However, they are one of the few backtest elements which accurately characterize time series!

•  Drawdowns and permanent capital loss may not appear in a backtest curve!

•  Don’t Trust Lines!

The future is like the past!

The future is not like the past!

Don’t Trust Lines: Lookahead Bias!

Executing this strategy is not realistic!

!Linear scale and

starting point are not!representative!

!…sure looks good

though!!

Curve Fitting: Do’s and Don’ts!

•  Don’t.!

Conclusion!

•  Question everything (is the backtest realistic?)!

•  Trust nothing (is the backtest representative?)!

•  Don’t be a turkey!!