National Instituteof Economic and Social Research

Combining forecast densities from VARs with uncertain instabilities

Anne Sofie Jore (Norges Bank)

James Mitchell (NIESR, London)

Shaun Vahey (Norges Bank, Melbourne Business School and RBNZ)

CIRANO Workshop: 5-6 Oct. 2007

• Clark and McCracken (2007) find that simple averages of VAR forecasts improve RMSE – Averaging circumvents unknown structural breaks or

“uncertain instabilities”

• This finding is reassuring for decision makers with quadratic loss functions– For more general but unknown loss functions, the

effectiveness of simple averages for VARs has not been studied

• We generalize CM to study the forecast densities produced by averaging VAR densities

Forecast Combination

The bottom-line• We find a substantial difference between the

accuracy of the combined forecast densities before and after the US Great Moderation

• In the evaluation period 1970-84 simple combinations perform reasonably well

• But over 1985-2005 equal weight density combination performs poorly. In contrast CM showed that RMSEs from simple averages performed well

• If greater weight is given to models that allow for the shifts in volatilities associated with the Great Moderation, predictive density accuracy improves considerably

Illustratation

• The rest of the paper provides more formal evidence that evidence-based weights deliver better density forecasts than equal weights

• To facilitate comparison with CM we use the same real-time data and estimate the same VAR models in output, prices and the short-term interest rate

• The models include both full-sample and rolling window VARs, first-differenced VARs, de-trended VARs and AR benchmarks– First differencing, de-trending and rolling window models may offer

some protection against structural breaks

– Combining models with varying degrees of adaptability to structural

breaks can help; see Pesaran and Timmermann (2007)

Density forecast combination

Decision Maker’s Expert Combination• Following Morris (1974, 1977) and Winkler (1981) we adopt

an “expert” combination approach• We define i=1,…,N experts, where each expert produces

one of the N VARs (and univariate benchmarks)• A decision maker (DM) combines the densities from two or

more experts based on the fit of the density forecasts over the Evaluation Period (DMEP)– The experts do not interact prior to combination by the DM

• The DM learns about the predictive densities by combining prior information with the evidence presented by the experts in the DMEP using Bayes rule

Linear opinion pool• The DM constructs the combined densities by a linear

opinion pool method (see Mitchell & Hall, 2005)

where are the forecast densities of expert i of variable using information set which given publication delays includes information dated and earlier

• are a set of non-negative weights that sum to unity • The linear opinion pool delivers a more flexible distribution

than each of the individual densities and accounts for model uncertainty

, ,1( ) ( | )

N

i iip y w g y I

,( | )ig y I

y ,iI 1

,iw

Methods of choosing wi

• Equal weights– The DM ignores the information in the data (the

evaluation period) and simply assigns an equal prior weight to each expert in all periods:

– We consider each of the pairwise combinations considered by CM (2007), and a combination taken across all experts

• De-trending (exponential smoothing) was found to work particularly well – accounts for non-stationarities

1/iw N

Methods of choosing wi (cont.)• Evidence-based weights

– The DM combines the information yielded from the fit of the experts’ forecast densities with prior information

• We assume the DM has non-informative priors over the experts

– We use the log score (logS) to measure the fit of the experts’ densities through the DMEP

– These predictive weights minimize the Kullback-Leibler Information Criterion (KLIC) ‘distance’ between the combined density forecast and the true but unknown density of

,

,

,1

exp ln ( | )

exp ln ( | )

i

i N

ii

g y Iw

g y I

y

Evaluation of forecast density combinations• For the last observation forecast the DM’s (post

realization) weights across the N models captures the fit of each expert’s density

• We also follow CM (2007) and evaluate the forecast densities over two evaluation periods:– 1970Q1-1984Q4 and 1985Q1-2005Q4

• Density forecasts are evaluated by testing whether the probability integral transforms of the forecast density with respect to the realization of the variable are uniform

The real-time US data

• We use the same real-time data as CM (2007) and like them run (stationary) VARs in output, inflation and the interest rate– We ignore any uncertainty associated with the trend

• The raw data are from the Federal Reserve Bank of Philadelphia's Real-Time Data Set for Macroeconomists (Croushore & Stark, 2001)

• All experts produce forecasts using the sequence of data vintages starting in 1965Q4 and ending in 2005Q4, with the raw data dating back to 1955

• To evaluate the forecasts the DM uses the second estimates as “final” data– hence DM combination using evidence-based weights introduces a

two-quarter lag. Like CM we examine robustness to this definition

GDP growth h=0Q: 1970-84

proportion KLIC LR (p-value) AD (95% cv=2.5)

univariate 0.004 0.778 0.613 univariate rolling 0.001 0.938 1.532 VAR(4) 0.068 0.017 4.482 VAR(4) rolling 0.073 0.013 8.418 DVAR(4) 0.003 0.813 1.339 DVAR(4) rolling 0.011 0.507 4.783 VAR(4) Inflation detrending 0.001 0.961 0.216 VAR(4) Inflation detrending rolling 0.006 0.717 3.652

EW (equal) 0.810 0.009 0.569 1.143

EB (logS) 0.905 0.004 0.765 1.017

avg AR(2), VAR(4) 0.571 0.007 0.621 0.893 avg AR(2), DVAR(4) 0.619 0.002 0.863 0.586 avg AR(2) VAR(4) infl detrend 0.667 0.000 0.999 0.296 avg VAR(4), rolling VAR(4) 0.143 0.062 0.016 4.814 avg AR(2), rolling AR(2) 0.190 0.002 0.869 0.672 avg rolling AR(2), rolling VAR(4) 0.143 0.007 0.630 1.386 avg rolling AR(2), rolling VAR(4) infl detrend 0.143 0.001 0.921 1.054

GDP growth h=0Q: 1985-2005

proportion KLIC LR (p-value) AD (95% cv=2.5)

univariate 0.349 0.000 7.733 univariate rolling 0.014 0.312 1.158 VAR(4) 0.275 0.000 7.064 VAR(4) rolling 0.082 0.001 5.643 DVAR(4) 0.192 0.000 4.942 DVAR(4) rolling 0.057 0.009 1.927 VAR(4) Inflation detrending 0.295 0.000 7.426 VAR(4) Inflation detrending rolling 0.024 0.139 1.707

EW (equal) 1.000 0.180 0.000 5.105

EB (logS) 1.000 0.041 0.026 2.047

avg AR(2), VAR(4) 0.429 0.315 0.000 7.700 avg AR(2), DVAR(4) 0.429 0.282 0.000 6.759 avg AR(2) VAR(4) infl detrend 0.429 0.323 0.000 8.134 avg VAR(4), rolling VAR(4) 0.714 0.183 0.000 6.292 avg AR(2), rolling AR(2) 0.952 0.152 0.000 3.928 avg rolling AR(2), rolling VAR(4) 1.000 0.053 0.009 2.781 avg rolling AR(2), rolling VAR(4) infl detrend 1.000 0.045 0.019 2.313

The DM’s post realization weights

1970-84 1985-2005 AR(2) 0.000 0.000 AR(BIC) 0.000 0.000 AR(2) rolling 0.000 0.072 VAR(4) 0.000 0.000 VAR(BIC) 0.001 0.000 VAR(4) rolling 0.000 0.006 VAR(BIC) rolling 0.000 0.013 AR(BIC) rolling 0.000 0.056 DVAR(4) 0.000 0.000 DVAR(BIC) 0.000 0.000 DVAR(4) rolling 0.000 0.468 DVAR(BIC) rolling 0.000 0.008 VAR(4) infl. detr. 0.000 0.000 VAR(BIC) infl. detr. 0.998 0.000 VAR(BIC) rolling +10 0.000 0.047 VAR(BIC) rolling +20 0.000 0.239 VAR(BIC) rolling +30 0.000 0.000 VAR(BIC) rolling -10 0.000 0.002 VAR(BIC) rolling -20 0.000 0.000 VAR(BIC) rolling -30 0.000 0.000 VAR(4) infl. detr. Rolling 0.000 0.090

Summarizing the results

• Whereas CM (2007) show that the RMSEs from simple averages can beat those from benchmark univariate models, we show that neither approach produces accurate forecast densities over the period spanning the Great Moderation

• Evidence-based (EB) weights provide more accurate density forecasts by favoring models that allow for the shifts in volatilities associated with the Great Moderation– EB weights work by essentially delivering a flexible non-linear

model

Conclusion

• We have re-visited the real-time application of CM (2007) but generalized to study density forecasts of GDP growth, inflation and the interest rate

• Future work:– Add in DSGE models– Add in models with breaks– Explore the possibility of getting improved VAR forecasts using first

release data (Corradi, Fernandez and Swanson, 2007) – work out predictive weights accounting for model uncertainty