to combine forecasts or to combine forecast models?

19
DR. DEVON BARROW DR. NIKOLAOS KOURENTZES 35 TH INTERNATIONAL SYMPOSIUM ON FORECASTING MARRIOTT RIVERSIDE CONVENTION CENTRE 22 – 24 JUNE 2015 To combine forecasts or forecast models (Parameters)

Upload: devon-k-barrow

Post on 18-Aug-2015

70 views

Category:

Data & Analytics


3 download

TRANSCRIPT

DR. DEVON BARROWDR. NIKOLAOS KOURENTZES

3 5 T H I N T E R N A T I O N A L S Y M P O S I U M O N F O R E C A S T I N GM A R R I O T T R I V E R S I D E C O N V E N T I O N C E N T R E

2 2 – 2 4 J U N E 2 0 1 5

To combine forecasts or forecast models (Parameters)

18/04/2023

2

Outline

To combine forecasts or forecast models

Forecast combination and model uncertaintyResearch questions

Experimental DesignResults

Conclusion

“Everything should be made as simple as possible, but no simpler.” Albert Einstein (Supposedly)

18/04/2023To combine forecasts or forecast models

3

Model uncertainty

Model building process Model formulation (or model specification) Model fitting ( or model estimation) … (Chatfield, 1995)

Sources of uncertainty Model structure Model parameter estimates Unexplained random variations (Draper et al., 1987; Hodges, 1987)

18/04/2023To combine forecasts or forecast models

4

Forecast combinations

Given multiple models Select a single (forecast) model Create multiple forecasts Take a simple average of all forecasts

Does combination work? Generally leads to improved accuracy (Stock and Watson, 2004 ;Fildes, Nikolopoulos et al.

2008) etc…

More robust and accurate than individual forecast (Newbold and Granger,74; Palm and Zellner,92) etc...

M3-Competition (Makridakis et al. 2000) simple average (Comb S-H-D) outperforms others

Nearly 50 years of forecast combination research focused on Combining forecasts Combining model parameters almost neglected

18/04/2023To combine forecasts or forecast models

5

Combinations and model uncertainty

Why combinations work? Usually the model form is unknown to the forecaster

i.e. ‘true model’ Data generating process may not be of a simple

functional form Things change with time e.g. seasonality and/or trend

may disappear, structural breaks Outliers and anomalies may distort structure A finite number of observations, often small samples Large effects are easier to identify than smaller

effects

18/04/2023To combine forecasts or forecast models

6

Outline

Forecast combination and model uncertaintyResearch questions

Experimental DesignResults

Conclusion

18/04/2023To combine forecasts or forecast models

7

Research questions

Bagged exponential smoothing (Christoph, Hyndman & Benitez, 2014)

Perform STL decomposition Bootstrapped residuals of the decomposition Recombine to obtain new series Estimate a model for each bootstrapped series

Shown to work well – especially for monthly data Combine forecasts from a family of closely related methods

Consequently the model estimated to be best can vary from data set to data set

To combine forecasts or forecast models (parameters)?

18/04/2023To combine forecasts or forecast models

8

Research questions

Nearly 50 years of research focused on Combining forecasts Combining model parameters – almost neglected

Bagged exponential smoothing (Christoph, Hyndman & Benitez, 2014)

Perform STL decomposition Bootstrapped residuals of the decomposition Recombine to obtain new series Estimate a model for each bootstrapped series

Shown to work well – especially for monthly data Combine forecasts from a family of closely related methods

Consequently the model estimated to be best can vary from data set to data set

To combine forecasts or forecast models (parameters)?

18/04/2023To combine forecasts or forecast models

9

Research questions

[image of bootstrap series] [Image of changing models]

18/04/2023To combine forecasts or forecast models

10

Research questions

RQ1: Are there benefits to be achieved from combining forecast model parameters rather than model forecasts themselves? RQ 1.1: Given the same forecast model structure? RQ 1.2: When the forecast model structure is allowed

to vary?

RQ2: How can differences in performance be explained?

18/04/2023To combine forecasts or forecast models

11

Outline

Forecast combination and model uncertaintyResearch question

Experimental DesignResults

Conclusion

18/04/2023To combine forecasts or forecast models

12

Experimental design - setup

Combinations Parameter combination Forecast combination

Combination methods Mean, Median, Mode

Forecast model generation Bootstrap Simulation

Conditions Model structure fixed Model structure varies

18/04/2023To combine forecasts or forecast models

13

Experimental design - setup

Family of exponential smoothing methodsModel uncertainty

A general class of models where the true model is a special case Models of different structures

Source: Hyndman et al. 2002 based on (Pegels ,1969 ; Gardner 1985)

18/04/2023To combine forecasts or forecast models

14

Experimental design - benchmarks

Automatic model selection based on AIC (Hyndman et al. 2002)

Bagged ETS (Christoph, Hyndman & Benitez, 2014)

Perform STL decomposition Bootstrapped residuals of the decomposition Recombine to obtain new series Estimate a model for each bootstrapped series

Points of comparison: Model selection versus combination Model structure varies across bootstrapped series Forecasts and not parameters are combined

Source: Hyndman et al. 2002 based on (Pegels ,1969 ; Gardner 1985)

18/04/2023To combine forecasts or forecast models

15

Experimental design - evaluation

Research Question 1: Empirical evaluation M3 Competition data Forecast accuracy using SMAPE and GMRAE

Research Question 2: Bias-variance decomposition

18/04/2023To combine forecasts or forecast models

16

Outline

Forecast combination and model uncertaintyResearch question

Experimental DesignResults

Conclusion

18/04/2023To combine forecasts or forecast models

17

Results

18/04/2023To combine forecasts or forecast models

18

Outline

Forecast combination and model uncertaintyResearch question

Experimental DesignResults

Conclusion

18/04/2023To combine forecasts or forecast models

19

Conclusion