practical portfolio optimizationfaculty.london.edu/avmiguel/ppo-iccopt-2013-07-24.pdf · practical...

Practical Portfolio Optimization

Victor DeMiguelProfessor of Management Science and Operations

London Business School

Based on joint research with

Lorenzo Garlappi Alberto Martin-Utrera Xiaoling MeiU. of British Columbia U. Carlos III de Madrid U. Carlos III de Madrid

Francisco J. Nogales Yuliya Plyakha Raman UppalU. Carlos III de Madrid Goethe University Frankfurt EDHEC Business School

Grigory VilkovGoethe University Frankfurt

International Conference on Continuous OptimizationJuly 30, 2013

“The motivation behind my dissertation was to applymathematics to the stock market” Harry Markowitz

“This is not a dissertation in economics, and we cannotgive you a PhD in economics for this” Milton Friedman

Mean-variance efficient frontier

I Efficient frontier: Investor concerned only about mean andvariance of returns chooses portfolio on efficient frontier.

-

6Expected

Return

Variance

..........................

........................

........................

........................

........................

.......................................................

.................................................................

...........................................................................

..................................................................................... ..............................................

3 / 70

Mean Variance portfolio optimization

minw

w>Σw︸ ︷︷ ︸variance

− w>µ︸︷︷︸mean

/γ

s.t. w ∈ C︸ ︷︷ ︸constraints

I w portfolio weight vector

I Σ covariance matrix of asset returns

I µ mean asset returns

I γ risk aversion parameter

4 / 70

Implementation: estimation and rebalancing

-

-�Estimation window

?

Next month

Portfolio for January 2009

TimeJan 98 Dec 08

5 / 70

-

-�Estimation window

?

Next month

Portfolio for January 2009

TimeJan 98 Dec 08

-

-�Estimation window

?

Next month

TimeFeb 98 Jan 09

Portfolio for February 2009

Will the portfolios for January 2009 and February 2009 be similar?

6 / 70

Problem: unstable portfolios

I The portfolios for consecutive months usually differgreatly:

I Unstable portfolios: Extreme weights that fluctuate a lot aswe rebalance the portfolio

I Why?

I Estimation error!!!

-

6Weight onrisky asset

Time

�����������CCCCCCCC��������BBBBBBB����������BBBBB@@

7 / 70

Problem: unstable portfolios

I The portfolios for consecutive months usually differgreatly:

I Unstable portfolios: Extreme weights that fluctuate a lot aswe rebalance the portfolio

I Why?

I Estimation error!!!

-

6Weight onrisky asset

Time

�����������CCCCCCCC��������BBBBBBB����������BBBBB@@

8 / 70

The 1/N paper

The 1/N paperDeMiguel, Garlappi, Uppal (RFS, 2009)

Objective: : Quantify impact of estimation error by comparingmean-variance and equal-weighted portfolio (1/N).

Why use the 1/N portfolio as a benchmark?

I Estimation-error free,I Simple but not simplistic:

I Does have some diversification, though not “optimally”diversified;

I With rebalancing =⇒ contrarian;I Without rebalancing =⇒ momentum.

I Ancient wisdom

I Rabbi Issac bar Aha (Talmud, 4th Century): Equal allocationA third in land, a third in merchandise, a third in cash.

I Investors use it, even nowadays.11 / 70

Thomson Reuters equal weighted commodity index

What we do

I Empirically: Compare fourteen portfolio rules to 1/Nacross seven datasets

I Sharpe ratio

SR =mean

std .dev .

I Analytically: Derive critical estimation windowlength for mean-variance strategy to outperform 1/N .

I Simulations: Extend analytical results to modelsdesigned to handle estimation error.

13 / 70

What we find

I Empirically: None of the fourteen portfolio modelsconsistently dominates 1/N across seven separatedatasets (SR and turnover).

I Analytically: Based on U.S. stock market data, criticalestimation window for sample-based mean variance(MV) to outperform 1/N is

I Approximately 3,000 months for 25-asset portfolio

I Approximately 6,000 months for 50-asset portfolio

I Simulations: Even models designed to handle estimationerror need unreasonably large estimation windows tooutperform 1/N .

14 / 70

Portfolios tested

I Bayesian portfolios

15 / 70

Historical return data

Sample mean

Historical return data

Sample mean

Mean prior distribution

Historical return data

Sample mean

Mean prior distribution Mean posterior

distribution

Bayes rule

Portfolios tested

I Bayesian portfolios

I Diffuse prior [Barry, 1974], [Klein and Bawa, 1976], [Brown, 1979],

I Informative empirical prior [Jorion, 1986]

I Prior belief on asset pricing model [Pastor, 2000] and

[Pastor and Stambaugh, 2000]

I Portfolios with moment restrictions

I Portfolios subject to shortsale constraints

I Optimal combinations of portfolios

19 / 70

Portfolios tested

I Bayesian portfolios

I Portfolios with moment restrictions

I Minimum-variance portfolio often outperforms mean-varianceportfolios; [Jagannathan and Ma, 2003],

I Value-weighted market portfolio optimal in a CAPM world,

I Missing-factor portfolio. [MacKinlay and Pastor, 2000] impose

Σ = νµµ> + σ2IN .

I Portfolios subject to shortsale constraints

I Optimal combinations of portfolios

20 / 70

Portfolios tested

I Bayesian portfolios

I Portfolios with moment restrictions

I Portfolios subject to shortsale constraints[Jagannathan and Ma, 2003] show they perform well inpractice.

I Optimal combinations of portfolios

21 / 70

Portfolios tested

I Bayesian portfolios

I Portfolios with moment restrictions

I Portfolios subject to shortsale constraints

I Optimal combinations of portfolios[Kan and Zhou, 2007]

wKZ = a wmean-variance + b wminimum-variance,

where a and b minimize portfolio loss.

22 / 70

Empirical results: Out-of-sample Sharpe ratios - I

Sector Indust. Inter’l Mkt/ FF FFStrategy Portf. Portf. Portf. SB/HL 1-fact. 4-fact.

1/N 0.19 0.14 0.13 0.22 0.16 0.18

mean var (in sample) 0.38 0.21 0.21 0.29 0.51 0.54

Classical approach that ignores estimation errormean variance 0.08 -0.04 -0.07 0.22 -0.07 0.00

Bayesian approach to estimation errorBayes Stein 0.08 -0.03 -0.05 0.25 -0.06 0.00data and model 0.14 -0.05 0.10 0.02 0.07 0.24

Moment restrictionsminimun variance 0.08 0.16 0.15 0.25 0.28 -0.02market portfolio 0.14 0.11 0.12 0.11 0.11 0.11missing factor 0.19 0.13 0.12 0.06 0.15 0.15

23 / 70

Why does minimum-variance portfolio outperformmean-variance portfolio?

I In sample, min-var portfolio has smallest expected return.

-

6Expected

Return

Variance

.........................

........................

.......................

.......................

........................

........................................................

..................................................................

.............................................................................

.......................................... .............................................in-sample frontier

24 / 70

Why does minimum-variance portfolio outperformmean-variance portfolio?

I In sample, min-var portfolio smallest expected return.

I Out of sample, minimum-variance portfolio has typicallyhighest Sharpe ratio; Jorion (1985, 1986, 1991).

I Estimation error in mean larger than in variance; Merton (1980).

I Jagannathan and Ma (2003): “estimation error in the sample mean is solarge that nothing much is lost in ignoring the mean altogether”.

-

6Expected

Return

Variance

.........................

........................

.......................

.......................

........................

........................................................

..................................................................

.............................................................................

.......................................... .............................................

........ .......... .............. ................. .................... ....................... .......................... ............................. ................................ ................................... ...................................... ......................................... ............................................

in-sample frontier

out-of-sample frontier

25 / 70

Mean variance portfolio returns are extreme

Jul−84 Jul−88 Sep−92 Nov−96 Jan−01 Apr−05 Jun−09−1.5

−1

−0.5

0

0.5

1

1.5

Dates

Por

tfolio

Ret

urns

Mean−VarianceEqually Weighted

26 / 70

Empirical results: Out-of-sample Sharpe ratios - II

Sector Indust. Inter’l Mkt/ FF FFStrategy Portf. Portf. Portf. SB/HL 1-fact. 4-fact.

1/N 0.19 0.14 0.13 0.22 0.16 0.18

mean var (in sample) 0.38 0.21 0.21 0.29 0.51 0.54

Shortsale constraintsmean variance 0.09 0.07 0.08 0.11 0.20 0.20Bayes Stein 0.11 0.08 0.08 0.15 0.20 0.21minimum variance 0.08 0.14 0.15 0.25 0.15 0.36

Optimal combinations of portfoliosmean & min-var 0.07 -0.03 0.06 0.25 -0.01 0.001/N & min-var 0.12 0.16 0.14 0.25 0.26 -0.02

27 / 70

Why do shortselling constraints help?

I Intuitively, they prevent extreme (large) positive and negativeweights in the portfolio.

I Theoretically, imposing shortselling constraints is likereducing the covariances assets that we would short;Jagannathan and Ma (2003).

Σ = Σ− λe> − eλ>

-

6Weight onrisky assetconstrained

Time

�����������CCCCCCCC��������BBBBBBB����������BBBBB@@

28 / 70

Results: Turnover relative to 1/N

Sector Indust. Inter’l Mkt/ FF FFStrategy Portf. Portf. Portf. SB/HL 1-fact. 4-fact.

1/N 3.05% 2.16% 2.93% 2.37% 1.62% 1.98%

Mean variance and Bayesian portfoliosmean variance 39 607479 4236 2.83 9408 2672Bayes Stein 22.41 10092 1760 1.85 10510 2783data and model 1.72 20468 45.24 76.85 762.83 32.46

Moment restrictionsminimum variance 6.54 21.65 7.30 1.11 45.47 6.83market portfolio 0 0 0 0 0 0missing factor 1.10 1.05 1.10 4.41 1.09 0.98

Shortsale constraintsmean variance 4.53 7.17 7.23 4.12 17.53 13.82Bayes Stein 3.64 7.22 6.10 3.65 17.32 13.07minimum variance 2.47 2.58 2.27 1.11 3.93 1.76

Optimal combinations of portfoliosmean & min-var 19.82 9943 819.30 2.61 3762 42871/N & min-var 4.82 15.66 4.24 1.11 34.10 6.80

29 / 70

Empirical results: Summary

I In-sample, mean-var strategy has highest Sharpe ratio.

I Out-of-sample

I 1/N does quite well in terms of Sharpe ratio and turnover.

I Sample-based mean-var strategy has worst Sharpe ratio.

I Bayesian policies typically do not out-perform 1/N.

I Constrained policies do better than unconstrained,but constraints alone do not help (need extra momentrestrictions).

I Min-var-constrained does well, but:I Sharpe Ratios and CEQ statistically indistinguishable from

1/N.

I Better than 1/N in only one dataset (20-size/bm portfolios).

I Turnover is 2-3 times higher than 1/N.

30 / 70

Beating 1/N

Beating 1/N

minw

w>Σw︸ ︷︷ ︸variance

− w>µ︸︷︷︸mean

/γ

s.t. w ∈ C︸ ︷︷ ︸constraints

Improve portfolio performance using

I. better covariance matrix,

II. better mean,

III. better constraints.

33 / 70

I. Better covariance matrix

To estimate covariance matrix better:

1. Use higher-frequency data

I more accurate estimates of covariance matrix; [Merton, 1980].

2. Use factor models

3. Use shrinkage estimators

34 / 70

I. Better covariance matrix

To estimate covariance matrix better:

1. Use higher-frequency data

2. Use factor models

I r︸︷︷︸returns

= a︸︷︷︸intercept

+ B︸︷︷︸slopes

f︸︷︷︸factors

+ε.

I more parsimonious estimates; [Chan et al., 1999].

3. Use shrinkage estimators

35 / 70

I. Better covariance matrix

To estimate covariance matrix better:

1. Use higher-frequency data

2. Use factor models

3. Use shrinkage estimators

I [Ledoit and Wolf, 2004] combine sample covariance and

identity

ΣLW = α Σ + (1− α)I .

36 / 70

Shrinkage estimators

ΣI

Distribution of sample covariance matrix estimator

Distribution of shrunkcovariance matrix estimator

Distribution of deterministiccovariance matrix estimator Bias

Variance Σ

Variance Σ

ΣLW

37 / 70

I. Better covariance matrix

To obtain better estimates of covariance matrix:

1. Use higher-frequency data

2. Use factor models

3. Use shrinkage estimators

4. Use robust optimization

I Goldfarb and Iyengar (MOR, 2003) and many others

minw

maxΣ∈U

w>Σw− w>µ/γ

s.t. w ∈ C

5. Use robust estimation

38 / 70

I. Better covariance matrix

To obtain better estimates of covariance matrix:

1. Use higher-frequency data

2. Use factor models

3. Use shrinkage estimators

4. Use robust optimization

5. Use robust estimation

39 / 70

Portfolio Selection with Robust EstimationDeMiguel and Nogales (Operations Research, 2009)

Deviation

.

..................................................

...............................................

...........................................

........................................

.....................................

..................................

..............................

...........................

........................

.....................

.................... ................... .................. ................. ................. ..............................................................................

........................

...........................

..............................

..................................

.....................................

........................................

...........................................

...............................................

..................................................

Variance

I Square function amplifies the impact of outliers

40 / 70

Portfolio Selection with Robust Estimation

Deviation

.

..................................................

...............................................

...........................................

........................................

.....................................

..................................

..............................

...........................

........................

.....................

.................... ................... .................. ................. ................. ..............................................................................

........................

...........................

..............................

..................................

.....................................

........................................

...........................................

...............................................

..................................................

�����������

@@

@@

@@@

@@

@@

Mad

I Absolute value function reduces impact of outliers

41 / 70

Portfolio Selection with Robust Estimation

Deviation

.

..................................................

...............................................

...........................................

........................................

.....................................

..................................

..............................

...........................

........................

.....................

....................

...................

.................. ................. ................. .....................................

....................

.....................

........................

...........................

..............................

..................................

.....................................

........................................

...........................................

...............................................

..................................................

�����������

@@

@@

@@@

@@

@@

...............

......

...................

.................. ................. ................. .....................................

....................@@

@@@

@@@

@@@

�����������

M-estimator

I M-estimators use smooth error function

42 / 70

Portfolio Selection with Robust Estimation

Deviation

.

..................................................

...............................................

...........................................

........................................

.....................................

..................................

..............................

...........................

........................

.....................

.................... ................... .................. ................. ................. ..............................................................................

........................

...........................

..............................

..................................

.....................................

........................................

...........................................

...............................................

..................................................

�����������

@@

@@

@@@

@@

@@

. .................... ................... .................. ................. ................. .........................................................

@@@

@@@

@@

@@@

�����������

.

....................................

.................................

..............................

...........................

.......................

.....................

.................... ................... .................. ................. ................. ..............................................................................

.......................

...........................

..............................

.................................

....................................

S-estimator

I S-estimators: the impact of outliers is bounded

43 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

4. Use option-implied information

44 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

4. Use option-implied information

45 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

minw

w>Σw−minµ∈U

w>µ/γ

s.t. w ∈ C

4. Use option-implied information

46 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

4. Use option-implied information

47 / 70

Improving Portfolio Selection UsingOption-Implied Volatility and Skewness

DeMiguel, Plyakha, Uppal, and Vilkov, forthcoming in JFQA

I Implied volatilities improvevolatility by 10-20%.

I Implied correlations do notimprove performance.

I Implied skewness andvolatility risk premiumproxy mean returns.

I Improve Sharpe ratioeven with moderatetransactions costs forweekly and monthlyrebalancing.

48 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

4. Use option-implied information

5. Use stock return serial dependence

49 / 70

Stock Return Serial Dependence and Out Of SamplePerformance, DeMiguel, Nogales, Uppal (2013)

I Stock return serialdependence

I Contrarian: “buy losersand sell winners”.

I Momentum: “buywinners and sell losers”.

I Can this be exploitedsystematically with manyassets?

I Yes, for transaction costsbelow 10 basis points.

50 / 70

II. Better mean

To obtain better estimates of mean:

1. Ignore means

2. Use Bayesian estimates

3. Use robust optimization

4. Use option-implied information

5. Use stock return serial dependence

6. Exploit anomalies: size, momentum, value

I “Expected Returns” by Antti Ilmanen.

51 / 70

III. Better constraints

To improve performance impose:

1. Shortsale constraints

2. Parametric portfolios

3. Norm constraints

52 / 70

III. Better constraints

To improve performance impose:

1. Shortsale constraints

2. Parametric portfolios[Brandt et al., 2009] impose constraint

w1

w2

...wN

=

wM

1wM

2...

wMN

+

me1 btm1 mom1

me2 btm2 mom2

......

...meN btmN momN

θme

θbtmθmom

,

me = market equity,btm = book-to-market ratio,

mom = momentum or average return over past 12 months.

3. Norm constraints

53 / 70

III. Better constraints

To improve performance impose:

1. Shortsale constraints

2. Parametric portfolios

3. Norm constraints

54 / 70

A Generalized Approach to Portfolio Optimization:Improving Performance by Constraining Portfolio Norms

DGNU (MS, 2009)

minw

w>Σw

s.t. w>e = 1

‖w‖ ≤ δ

I Nests 1/N, shrinkage covariance matrix of[Ledoit and Wolf, 2004], and shortsale constraints,

I Diversification: 2-norm,

I Leverage constraint: 1-norm.55 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:

I fixed costs and cardinality constraints,

I value at risk.

I Approximate dynamic programming:

I dynamic portfolio choice,

I transaction costs.

I Calibration, calibration, calibration:

57 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:

I fixed costs and cardinality constraints,

I value at risk.

I Approximate dynamic programming:

I dynamic portfolio choice,

I transaction costs.

I Calibration, calibration, calibration:

58 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:

I fixed costs and cardinality constraints,

I value at risk.

I Approximate dynamic programming:

I dynamic portfolio choice,

I transaction costs.

I Calibration, calibration, calibration:

59 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:

I fixed costs and cardinality constraints,

I value at risk.

I Approximate dynamic programming:

I dynamic portfolio choice,

I transaction costs.

I Calibration, calibration, calibration:

60 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:

I fixed costs and cardinality constraints,

I value at risk.

I Approximate dynamic programming:

I dynamic portfolio choice,

I transaction costs.

I Calibration, calibration, calibration:

61 / 70

Same dog, different collar?Devil Is in One Detail: Calibration

Constraints Robust

optimization

Bayesian

portfolios

Shrinkage

estimators

Gotoh &

Takeda CMS 2011

DGNU MS 2009

Goldfarb

Iyengar MOR 2003

Caramanis

Mannor

Xu (2011)

Jorion JFQA 1986

Jaganathan

Ma JoF, 2003

DGNU MS 2009

62 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:I fixed costs and cardinality constraints,I value at Risk.

I Approximate dynamic programming:I dynamic portfolio choice,I transaction costs.

I Calibration, calibration, calibration:I “Size Matters: Optimal Calibration of Shrinkage Estimators

for Portfolio Selection”, D., Martin-Utrera, Nogales, (JB&F,

2012):I Choose criterion carefully, get nonparametric.I “Parameter Uncertainty in Multiperiod Portfolio Optimization

with Transaction Costs”, D., Martin-Utrera, Nogales.

I Shrink both target portfolio and trading rate.

I Statistics/big data/real-time estimation and optimization:I high-frequency trading.

63 / 70

Help wanted

I Optimization can make a difference in portfolio selection.

I Research opportunities

I Integer programming:I fixed costs and cardinality constraints,I value at Risk.

I Approximate dynamic programming:I dynamic portfolio choice,I transaction costs.

I Calibration, calibration, calibration:I “Size Matters: Optimal Calibration of Shrinkage Estimators

for Portfolio Selection”, D., Martin-Utrera, Nogales, (JB&F,

2012):I Choose criterion carefully, get nonparametric.I “Parameter Uncertainty in Multiperiod Portfolio Optimization

with Transaction Costs”, D., Martin-Utrera, Nogales.

I Shrink both target portfolio and trading rate.

I Statistics/big data/real-time estimation and optimization:I high-frequency trading.

64 / 70

High frequency trading

65 / 70

High frequency trading

66 / 70

Thank you

Victor DeMiguelhttp://www.london.edu/avmiguel/

67 / 70