Decision 411: Class 6Decision 411: Class 6

Fitting regression models to time series data Fitting regression models to time series data

Economic interpretation of coefficientsEconomic interpretation of coefficients

How to model seasonality with regressionHow to model seasonality with regression

LogLog--log (constant elasticity) modelslog (constant elasticity) models

Automatic stepwise variable selectionAutomatic stepwise variable selection

What will be on Tuesday’s quiz?What will be on Tuesday’s quiz?

Quiz will be openQuiz will be open--book, openbook, open--notes, 1st hournotes, 1st hour

Manual calculation of forecast and confidence limits Manual calculation of forecast and confidence limits for mean or RW modelfor mean or RW model

Tentative model identification based on exploratory Tentative model identification based on exploratory plotsplots

Interpretation of output: model comparisons, Interpretation of output: model comparisons, insignificant variables, diagnostic testsinsignificant variables, diagnostic tests

Criteria for choosing the “best” modelCriteria for choosing the “best” model

Regression analysis: recapRegression analysis: recapA A simplesimple regression regression model merely fits a model merely fits a straight linestraight line to a to a scatterplotscatterplot of of YY (the (the “dependent” variable) “dependent” variable) versus versus XX (the (the “independent” variable).“independent” variable).

The correlation between X and Y, together with their means and standard deviations, determines the slope and intercept of the regression line.

40

45

50

55

60

65

14 16 18 20 22 24 26

X

Y

Multiple regression equationMultiple regression equationGeneral multiple regression equation for General multiple regression equation for predicting a “dependent” variable predicting a “dependent” variable YY from from “independent” variables“independent” variables XX11, , XX22, …, , …, XXkk::

0 1 1 2 2ˆ ˆ ˆ ˆˆ ...t t t k ktY X X Xβ β β β= + + + +

Constant term is the “baseline” that would be obtained if all X’s were zero at the same time (if that is logically possible). More generally it just moves the regression line up or down to hit the center of the Y data.

The rest of the prediction equation consists of a weighted sum of the X’s. Hence the predicted pattern of Y is a weighted sum of the patterns in the X’s. In general, the weights (coefficients) may be positive or negative.

If If YY and and XX are are time seriestime series, the , the time dimensiontime dimension in the in the data ought to be taken into account. data ought to be taken into account.

When the forecasts for Yare plotted versus time, they no longer lie on a straight line: they are just a rescaled version of X!

This is the essence of a linear model.

If there’s more than one Xvariable, the predictions for Y are a sum of rescaled copies of the X’s, plus the intercept. (The scaling factors may be positive or negative.)0

10

20

30

40

50

60

70

1 2 3 4 5 6

XYFCST

VariablesY

Multiple X Y Plot

0 4 8 12 16 20T

0

0.4

0.8

1.2

1.6

When plotted versus time, the “dependent” variable When plotted versus time, the “dependent” variable to be predicted (Y) might look something like this:to be predicted (Y) might look something like this:

Multiple X Y Plot

0 4 8 12 16 20T

0

0.2

0.4

0.6

0.8

1

1.2 VariablesX

An “independent” variable (X) to be used for An “independent” variable (X) to be used for predicting Y might itself be another random variable:predicting Y might itself be another random variable:

p

0 4 8 12 16 20T

0

4

8

12

16

20

……or it could be a time trend (“slope”) variable:or it could be a time trend (“slope”) variable:

A time trend variable actually serves to “A time trend variable actually serves to “detrenddetrend” ” Y Y and and all all the the XX’s prior to estimating the other coefficients of the model, ’s prior to estimating the other coefficients of the model, i.e., it corrects for i.e., it corrects for differences in trenddifferences in trend among all the variables.among all the variables.

p

0 4 8 12 16 20T

0

2

4

6

8

10

……or X could be a or X could be a change in trend change in trend at a specific at a specific point in time:point in time:

p

0 4 8 12 16 20T

0

50

100

150

200

250

300

……or a nonlinear (e.g. quadratic) curve:or a nonlinear (e.g. quadratic) curve:

p

0 4 8 12 16 20T

-0.5

0

0.5

1

1.5

……or a dummy variable for occasional or periodic or a dummy variable for occasional or periodic events (e.g., seasons of the year):events (e.g., seasons of the year):

p

0 4 8 12 16 20T

-0.5

0

0.5

1

1.5

……or a dummy variable for a stepor a dummy variable for a step--change at a change at a specific point in time:specific point in time:

……or a lagged value of the dependent variable:or a lagged value of the dependent variable:

VariablesYLAG(Y,1)LAG(Y,2)

0 4 8 12 16 20T

0

0.4

0.8

1.2

1.6

……or a lagged value of an independent variable:or a lagged value of an independent variable:

VariablesXLAG(X,1)LAG(X,2)

Multiple X Y Plot

0 4 8 12 16 20T

0

0.2

0.4

0.6

0.8

1

1.2

How to fit regression modelsHow to fit regression modelsIn principle, all you need to do is find the In principle, all you need to do is find the rightrightindependent independent variable(svariable(s) for predicting ) for predicting youryourdependent variable.dependent variable.

In practice, you may also need to In practice, you may also need to transformtransform the the variables to improve the linearity of the relationship variables to improve the linearity of the relationship or the distribution of the errors (e.g., by lagging, or the distribution of the errors (e.g., by lagging, logging, deflating, differencing, multiplying or logging, deflating, differencing, multiplying or dividing variables, etc. etc.).dividing variables, etc. etc.).

You may also need to select the appropriate You may also need to select the appropriate amount of past data to use in model fitting.amount of past data to use in model fitting.

ModelModel--fitting stepsfitting steps1. Collect data for dependent & independent variables1. Collect data for dependent & independent variables 2. Identify potentially useful data transformations2. Identify potentially useful data transformations

3. Fit preliminary models; ideally “hold out” data for 3. Fit preliminary models; ideally “hold out” data for outout--ofof--sample testing.sample testing.

4. Screen out insignificant variables & look for other 4. Screen out insignificant variables & look for other ways to simplify or fineways to simplify or fine--tune.tune.

5. Check residual diagnostics to test assumptions5. Check residual diagnostics to test assumptions6. Compare performance against simpler models 6. Compare performance against simpler models

(e.g., random walk or other time series models that (e.g., random walk or other time series models that do not use “exogenous” X variables).do not use “exogenous” X variables).

What’s the bottom line?What’s the bottom line?RR--squared is squared is notnot the bottom linethe bottom line——it may not even be it may not even be comparable among models fitted with different data comparable among models fitted with different data samples and/or transformations, and there is no samples and/or transformations, and there is no universal standard for “good”.universal standard for “good”.

Residual diagnostics & Residual diagnostics & tt--stats are stats are notnot the bottom the bottom line, just “red flags” that may wave to indicate line, just “red flags” that may wave to indicate problems with model assumptions.problems with model assumptions.

Error measures (RMSE, MAPE, MAE) Error measures (RMSE, MAPE, MAE) areare the bottom the bottom lineline——when compared in the when compared in the same unitssame units——provided provided that they can be that they can be trustedtrusted, i.e., provided that model , i.e., provided that model assumptions appear to be valid.assumptions appear to be valid.

Simplicity & sound logic are also important: can you Simplicity & sound logic are also important: can you explain or sell the model to your boss or client?explain or sell the model to your boss or client?

OverOver-- and underand under--fittingfittingIf too few potential If too few potential regressorsregressors are considered, there are considered, there may be may be omitted variablesomitted variables whose effects are not whose effects are not captured or which will “load” onto other variables captured or which will “load” onto other variables (proxy effects).(proxy effects).If too many potential If too many potential regressorsregressors are considered, there are considered, there are dangers of are dangers of overover--fittingfitting from too much data mining from too much data mining (“spurious” (“spurious” regressorsregressors may be found).may be found).In either case, the model will predict the future less In either case, the model will predict the future less well than it fitted the past data.well than it fitted the past data.You need to exercise judgment to “preYou need to exercise judgment to “pre--screen” screen” potential potential regressorsregressors for relevance. Think before you for relevance. Think before you compute!compute!

Tools for identifying potential Tools for identifying potential regressorsregressorsScatterplotScatterplot matricesmatrices and and correlation matricescorrelation matrices may may help to identify variables related to help to identify variables related to YY..

ScatterplotsScatterplots also indicate whether “outliers” are also indicate whether “outliers” are present* and whether present* and whether nonlinear transformationsnonlinear transformationsmay be useful.may be useful.

Autocorrelation plotsAutocorrelation plots show whether lags of show whether lags of YY may may be useful as be useful as regressorsregressors..

CrosscorrelationCrosscorrelation plotsplots show whether lags of show whether lags of XX’s ’s may be useful as may be useful as regressorsregressors..

*Outliers should not be removed from the analysis just because they are outliers, but they should be given special attention: are they due to data entry errors, weird events that will not be repeated, or are they useful “natural experiments”?

Example of causal modeling: 3 years of monthly Example of causal modeling: 3 years of monthly sales and advertising data for a weightsales and advertising data for a weight--loss loss

product (unit sales, $ advertising)product (unit sales, $ advertising)

Index

VariablesSalesAdvertising

0 10 20 30 4012

17

22

27

32

37

0

20

40

60

80

Time series plot shows somewhatTime series plot shows somewhat--aligned peaks and valleys…aligned peaks and valleys…

……but what is the “bang but what is the “bang for the buck”?for the buck”?

Plot of Sales vs Adv

0 20 40 60 80Adv

12

17

22

27

32

37

Sale

s

XX--Y Y scatterplotscatterplot shows a positive, roughly linear shows a positive, roughly linear relationship, so let’s estimate the slope via relationship, so let’s estimate the slope via simple regression…simple regression…

??

Simple regression of sales on Simple regression of sales on advertisingadvertising

Simple regression suggests that $1 advertising yields additionalSimple regression suggests that $1 advertising yields additional0.208 units sold, but autocorrelations and residual0.208 units sold, but autocorrelations and residual--vsvs--time plot are time plot are bad. Perhaps lagged variables should be considered?bad. Perhaps lagged variables should be considered?

50% confidence limits for predictions were 50% confidence limits for predictions were selected via “pane options” for interval plotselected via “pane options” for interval plot

Which plots to look at?Which plots to look at?When fitting time series, the 3 most interesting plots When fitting time series, the 3 most interesting plots in the Multiple Regression procedure are usually:in the Multiple Regression procedure are usually:

1.1. Residuals vs. predicted valuesResiduals vs. predicted values (ideally the patterns (ideally the patterns are random and errors have the same variance for are random and errors have the same variance for small or large predictions)small or large predictions)

2.2. Residuals versus row number Residuals versus row number (ideally the pattern (ideally the pattern is random and does not show evidence of is random and does not show evidence of autocorrelation or autocorrelation or heteroscedasticityheteroscedasticity))

3.3. “Interval plots” “Interval plots” (data and forecasts plotted versus (data and forecasts plotted versus row number, or versus independent variables, with row number, or versus independent variables, with optional confidence limits superimposed)optional confidence limits superimposed)

Tools for identifying useful lags: autoTools for identifying useful lags: auto-- and and crosscross--correlations of sales and advertisingcorrelations of sales and advertising

Time Series/Descriptive Time Series/Descriptive Methods procedure Methods procedure shows that Sales has a shows that Sales has a significant autocorrelation significant autocorrelation at lag 1…at lag 1…

…… and the crossand the cross--correlation of between correlation of between sales and advertising is sales and advertising is significant at lag 1 as significant at lag 1 as well as lag 0.well as lag 0.

First try adding lagged advertising First try adding lagged advertising to modelto model

Updated resultsUpdated results

Much better error stats! This model suggests that the effect ofMuch better error stats! This model suggests that the effect ofadvertising carries over into two periods with a total impact ofadvertising carries over into two periods with a total impact of0.142+0.167 = 0.308 unit per $. But residuals have an upward 0.142+0.167 = 0.308 unit per $. But residuals have an upward trend, even worse autocorrelation. Add lagged sales next?trend, even worse autocorrelation. Add lagged sales next?

After adding lagged Sales, autocorrelations now look much betterAfter adding lagged Sales, autocorrelations now look much better, , and error stats are further improved. The economic interpretatiand error stats are further improved. The economic interpretation on of this model is a bit trickier. The sum of advertising coefficof this model is a bit trickier. The sum of advertising coefficients is ients is only 0.227, but this is amplified by the autoregressive sales faonly 0.227, but this is amplified by the autoregressive sales factor. ctor. The total impact is 0.227(1 + 0.364 + 0.364The total impact is 0.227(1 + 0.364 + 0.3642 2 + 0.364+ 0.3643 3 + …) + …) ≈≈ 0.36.0.36.

Where did that formula come from??!!Where did that formula come from??!!Adding lag(Sales,1) to the model implies that Adding lag(Sales,1) to the model implies that increases in Sales have “momentum” of their own.increases in Sales have “momentum” of their own.Its coefficient of 0.364 evidently means that boosting Its coefficient of 0.364 evidently means that boosting sales by 1 unit this period yields a further boost of sales by 1 unit this period yields a further boost of 0.364 in the next period (independent of this period’s 0.364 in the next period (independent of this period’s or next period’s advertising!), which in turn boosts or next period’s advertising!), which in turn boosts sales 2 periods ahead by 0.364sales 2 periods ahead by 0.36422, etc., etc.Hence the direct impact of an advertising $, which is Hence the direct impact of an advertising $, which is estimated to be 0.207 over two periods, gets amplified estimated to be 0.207 over two periods, gets amplified by a factor of 1 + 0.364 + 0.364by a factor of 1 + 0.364 + 0.36422 + 0.364+ 0.36433 + ...+ ...= 1/(1= 1/(1−−0.364) = 1.57 by the geometric series formula.0.364) = 1.57 by the geometric series formula.Hence the direct effect plus the momentum effect is a Hence the direct effect plus the momentum effect is a factor of 0.207 x 1.57 factor of 0.207 x 1.57 ≈≈ 0.36 units of sales per $ adv.0.36 units of sales per $ adv.

What’s the “real” bottom line?What’s the “real” bottom line?Different regression models may yield different Different regression models may yield different estimates of the economic impact of decisions, estimates of the economic impact of decisions, based on different underlying assumptions about based on different underlying assumptions about cause and effectcause and effect——e.g., lagged response, longe.g., lagged response, long--term term momentum, etc.momentum, etc.

The last two models suggest that the economic The last two models suggest that the economic impact of advertising lies somewhere between 0.30 impact of advertising lies somewhere between 0.30 and 0.36 unit per $.and 0.36 unit per $.

Note: rather than adding lagged sales to the model, we could have added one or two more lags of advertising as another way to capture effects of advertising that extend more than one period into the future. When lag(Advertising,2) is added rather than lag(Sales,1), it turns out to be marginally significant (t=1.6), and the sum of advertising coefficients rises to 0.35, about the same as above.

The same models can be fitted in the Forecasting The same models can be fitted in the Forecasting procedure: Mean model + Regression variablesprocedure: Mean model + Regression variables

Note: to get exactly the same results for these Note: to get exactly the same results for these laggedlagged--variable models as in the Multiple variable models as in the Multiple Regression procedure, you must deRegression procedure, you must de--select the first select the first row (by using Index >1 in the “Select” field on the row (by using Index >1 in the “Select” field on the input panel), otherwise input panel), otherwise StatgraphicsStatgraphics will try to will try to backforecastbackforecast the missing lagged values in row 1,the missing lagged values in row 1,

This yields headThis yields head--toto--head model comparisons:head model comparisons:

Also, the simple Also, the simple regression results are regression results are slightly different from slightly different from the original ones, the original ones, because the first data because the first data row was excluded row was excluded from all models for from all models for these comparisons.these comparisons.

A random walk model A random walk model has been included as has been included as a reference pointa reference point——note that it is as good note that it is as good as the original simple as the original simple regression.regression.

When using the Forecasting procedure to fit and compare When using the Forecasting procedure to fit and compare regression models, you cannot request forecasts for future regression models, you cannot request forecasts for future time periods unless future data is available for the time periods unless future data is available for the regressorsregressors. . This is an inherent limitation of regression forecasting models,This is an inherent limitation of regression forecasting models,in comparison to purely extrapolative time series models.in comparison to purely extrapolative time series models.

Dummy variablesDummy variables

A dummy variable is an independent variable A dummy variable is an independent variable whose values are all 1’s and 0’s, indicating the whose values are all 1’s and 0’s, indicating the presence or absence of some condition.presence or absence of some condition.

The estimated coefficient of the dummy variable is a The estimated coefficient of the dummy variable is a constant to be added to the forecast when the constant to be added to the forecast when the condition is “present”condition is “present”

Dummy variables are often used to model seasonal Dummy variables are often used to model seasonal effects as well as the effects of unusual events or effects as well as the effects of unusual events or structural changesstructural changes

Modeling seasonality with regressionModeling seasonality with regression

Suppose Suppose YY is a is a seasonalseasonal time series that is correlated time series that is correlated with some other “with some other “XX” variables” variables

Examples: advertising, promotions, prices, Examples: advertising, promotions, prices, competition, interest rates, economic indicators…competition, interest rates, economic indicators…

One modeling approach would be to seasonally adjust One modeling approach would be to seasonally adjust YY before fitting a regression model.before fitting a regression model.

However...However...

Potential problemsPotential problems

If the seasonal indices are estimated from the data If the seasonal indices are estimated from the data without “controlling” for effects of other without “controlling” for effects of other nonseasonalnonseasonal variables, the seasonal indices could variables, the seasonal indices could be in error and/or data could be be in error and/or data could be overfittedoverfitted..

If the seasonal indices are estimated from other If the seasonal indices are estimated from other (e.g., more highly aggregated data), then seasonal (e.g., more highly aggregated data), then seasonal adjustment of the data may distort the effects of the adjustment of the data may distort the effects of the other variables if those effects are really “constant” other variables if those effects are really “constant” over time.over time.

Possible solutionsPossible solutions

Use Use seasonal dummy variablesseasonal dummy variables as additional as additional regressorsregressors to estimate the seasonal effects while to estimate the seasonal effects while controlling for other variables.controlling for other variables.

Use an externallyUse an externally--supplied supplied seasonal indexseasonal index as an as an additional additional regressorregressor..

Use Use seasonal lags and/or seasonal differencesseasonal lags and/or seasonal differences, i.e., , i.e., base this period’s forecast on what happened one base this period’s forecast on what happened one year ago, not one period ago.year ago, not one period ago.

Additive or multiplicative?Additive or multiplicative?

Regression models are inherently Regression models are inherently additiveadditivemodels, therefore coefficients of seasonal dummy models, therefore coefficients of seasonal dummy variables represent an additive seasonal pattern.variables represent an additive seasonal pattern.

If the seasonal pattern is If the seasonal pattern is multiplicativemultiplicative, a log , a log transformation may be helpful: then the transformation may be helpful: then the coefficients of the seasonal dummies can be coefficients of the seasonal dummies can be converted to equivalent multiplicative seasonal converted to equivalent multiplicative seasonal factors.factors.

Gap revisitedGap revisited

For purposes of illustration, we’ll look at trendFor purposes of illustration, we’ll look at trend--line line and curveand curve--fitting models combined with seasonal fitting models combined with seasonal dummy variables.dummy variables.

We’ll also see how a dummy variable can be used to We’ll also see how a dummy variable can be used to model a trend change (“kink”) in a series.model a trend change (“kink”) in a series.

The variable NETSALES consists of quarterly net The variable NETSALES consists of quarterly net sales in $1000’s.sales in $1000’s.

Time Series Plot for NETSALES

Q3/97 Q3/99 Q3/01 Q3/03 Q3/05 Q3/0713

23

33

43

53(X 1.E5)

NE

TSA

LES

Gap data file: the QUARTER variable will be used to construct sGap data file: the QUARTER variable will be used to construct seasonal easonal dummy variables ondummy variables on--thethe--fly: the expression QUARTER=1 is a variable with fly: the expression QUARTER=1 is a variable with a 1 in every 1a 1 in every 1stst quarter and 0’s elsewhere, i.e., a dummy variable for the 1quarter and 0’s elsewhere, i.e., a dummy variable for the 1stst

quarter. An estimated regression coefficient for QUARTER=1 willquarter. An estimated regression coefficient for QUARTER=1 will be a be a constant to be added to the forecast in every 1st quarter. Simiconstant to be added to the forecast in every 1st quarter. Similarly, larly, QUARTER=2 and QUARTER=3 will be the dummies for the 2QUARTER=2 and QUARTER=3 will be the dummies for the 2ndnd and 3and 3rdrd

quarters. (We won’t use a 4quarters. (We won’t use a 4thth quarter dummyquarter dummy——the coefficients of the others the coefficients of the others will measure changes relative to the 4will measure changes relative to the 4thth quarter.)quarter.)

To get regression forecasts for To get regression forecasts for future time periods:future time periods:

The QUARTER variable has been extended 12 quarters into the futuThe QUARTER variable has been extended 12 quarters into the future, re, because it will be used to construct dummy variables in the regrbecause it will be used to construct dummy variables in the regressions, and essions, and regression models can’t generate forecasts for the future unlessregression models can’t generate forecasts for the future unless future values future values for the all the independent variables are available.for the all the independent variables are available.

Linear trend + seasonal dummiesLinear trend + seasonal dummies

Obviously not a good fitObviously not a good fit——does not respond to the general economic does not respond to the general economic downturn in 2001downturn in 2001--2002 and the flattening out of sales growth afterward, so 2002 and the flattening out of sales growth afterward, so residual time series and autocorrelation plots are very bad. Thresidual time series and autocorrelation plots are very bad. The coefficient e coefficient of QUARTER=1 is of QUARTER=1 is --1,202,000 (in $1000’s), hence 11,202,000 (in $1000’s), hence 1stst quarter sales are quarter sales are predicted to be $1,202M less than 4predicted to be $1,202M less than 4thth quarter sales, other things equal. For quarter sales, other things equal. For later reference, note that RMSE=$383M and MAPE=10% for this modlater reference, note that RMSE=$383M and MAPE=10% for this model.el.

The straightThe straight--line trend of this model is clearly inappropriate. line trend of this model is clearly inappropriate. The coefficient of t is the estimated longThe coefficient of t is the estimated long--term trend: term trend: $56M per quarter.$56M per quarter.

Time Sequence Plot for NETSALESLinear trend = -7.574E6 + 5.602E4 t + 3 regressors

Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/120

2

4

6

8(X 1.E6)

NET

SALE

S

The residual plot shows The residual plot shows showsshows a “kink” (change in trend) at Q4/00a “kink” (change in trend) at Q4/00

Residual Plot for adjusted NETSALESLinear trend = -7.574E6 + 5.602E4 t + 3 regressors

Q2/97 Q2/00 Q2/03 Q2/06 Q2/09-11

-7

-3

1

5

9(X 1.E5)

Res

idua

l

Let’s create a new variable called TRENDCHANGE whose Let’s create a new variable called TRENDCHANGE whose value is zero up to row 15 and which “ramps up” (1, 2, 3, …) value is zero up to row 15 and which “ramps up” (1, 2, 3, …) afterward. The estimated coefficient of this variable will afterward. The estimated coefficient of this variable will represent a change in the trend beginning at row 16.represent a change in the trend beginning at row 16.

Here are the results of adding TRENDCHANGE to the regression. THere are the results of adding TRENDCHANGE to the regression. The fit is he fit is dramatically improved: RMSE has dropped from $383M to $185M and dramatically improved: RMSE has dropped from $383M to $185M and MAPE MAPE has dropped from 10% to 4.3% for this model. Residual time serhas dropped from 10% to 4.3% for this model. Residual time series plot and ies plot and autocorrelation plot are still not great because the dip in 2001autocorrelation plot are still not great because the dip in 2001--2002 still has not 2002 still has not captured accurately.captured accurately.

Time Sequence Plot for NETSALESLinear trend = -2.49E7 + 1.434E5 t + 4 regressors

Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/120

1

2

3

4

5

6(X 1.E6)

NE

TSA

LES

The fit to the flatter recent trend has been much improved. TheThe fit to the flatter recent trend has been much improved. There re is still a slight upward trend in the longis still a slight upward trend in the long--range forecasts. The range forecasts. The coefficient of t, which is now $143M per quarter, is the estimatcoefficient of t, which is now $143M per quarter, is the estimated ed trend up to Q4/00, at which point the TRENDCHANGE variable trend up to Q4/00, at which point the TRENDCHANGE variable kicks in. Note the much tighter confidence intervals due to thekicks in. Note the much tighter confidence intervals due to thereduction in RMSE.reduction in RMSE.

The coefficient of The coefficient of TRENDCHANGE is TRENDCHANGE is --$122M $122M per quarter, so the estimated per quarter, so the estimated overall trend since Q4/00 is overall trend since Q4/00 is $144M $144M -- $122M = $22M per $122M = $22M per quarter. This is the slope of quarter. This is the slope of the longthe long--range forecasts on range forecasts on the forecast plot.the forecast plot.

Time Series Plot for adjusted NETSALES

Q2/97 Q2/00 Q2/03 Q2/06 Q2/0914

14.3

14.6

14.9

15.2

15.5

adju

sted

NE

TSA

LES

The preceding models assumed an The preceding models assumed an additiveadditive seasonal seasonal pattern, and the coefficients of the quarterly dummies pattern, and the coefficients of the quarterly dummies were the additive seasonal indices. were the additive seasonal indices. Alternatively,weAlternatively,wecould estimate could estimate multiplicativemultiplicative seasonal indices by fitting seasonal indices by fitting the same model with a the same model with a natural lognatural log transformation. transformation. Here is a plot of the logged data. The seasonal pattern Here is a plot of the logged data. The seasonal pattern does look more additive in these terms, and the trenddoes look more additive in these terms, and the trend--change at Q4/00 is still evident.change at Q4/00 is still evident.

This model assumes that sales This model assumes that sales grew at a constant percentage rate grew at a constant percentage rate up to Q4/00 and at a different up to Q4/00 and at a different percentage rate afterward. The percentage rate afterward. The slope coefficient of 0.056 is the slope coefficient of 0.056 is the prior growth rate (5.6% per prior growth rate (5.6% per quarter), and the TRENDCHANGE quarter), and the TRENDCHANGE coefficient of coefficient of −−0.051 is the change 0.051 is the change in the growth rate after Q4/00 in the growth rate after Q4/00 ((minusminus 5.1%). Hence the 5.1%). Hence the estimated growth rate after Q4/00 is estimated growth rate after Q4/00 is 5.6% 5.6% -- 5.1% = 0.5% per quarter.5.1% = 0.5% per quarter.

The error stats of this model are slight better than those of thThe error stats of this model are slight better than those of the e unloggedunlogged model (MAPE of 4.0% here vs. 4.3% earlier), although model (MAPE of 4.0% here vs. 4.3% earlier), although the residualthe residual--vsvs--time and residual autocorrelation plots don’t look time and residual autocorrelation plots don’t look quite as good. They are not the bottom line, though. Here theyquite as good. They are not the bottom line, though. Here theyindicate more room for improvement via some kind of fineindicate more room for improvement via some kind of fine--tuning.tuning.

Tournament results: the Tournament results: the unloggedunlogged and logged and logged regression models with regression models with the trend change the trend change (Models B and C) (Models B and C) compare favorably with compare favorably with Holt’s and Winters’ Holt’s and Winters’ models, despite the models, despite the autocorrelation in the autocorrelation in the residuals.residuals.

Equation of the logged modelEquation of the logged modelIn a model with a natural log transformation, In a model with a natural log transformation, conversion of the forecasts back to original units and conversion of the forecasts back to original units and interpreting the coefficients requires “interpreting the coefficients requires “unloggingunlogging” by ” by applying the EXP function.applying the EXP function.

In this model, the In this model, the unloggedunlogged forecasts have the forecasts have the equation equation f(tf(t) = EXP(3.98 + 0.056 t)) = EXP(3.98 + 0.056 t) up to Q4/00 and up to Q4/00 and

f(tf(t) = EXP(3.98 + 0.005 t)) = EXP(3.98 + 0.005 t) afterward,afterward,before the seasonal terms are factored in.before the seasonal terms are factored in.

By By EXP’ingEXP’ing the coefficients of the dummy variables, the coefficients of the dummy variables, we obtain multiplicative seasonal indices relative to we obtain multiplicative seasonal indices relative to Q4=100.Q4=100.

Seasonal indicesSeasonal indicesQuarter 1 index: EXP(Quarter 1 index: EXP(--0.335) = 71.5%0.335) = 71.5%Quarter 2 index: EXP(Quarter 2 index: EXP(--0.303) = 73.9%0.303) = 73.9%

Quarter 3 index: EXP(Quarter 3 index: EXP(--0.206) = 81.4%0.206) = 81.4%

Estimated coefficients of the dummy variables in logged model:

Estimated multiplicative indices from seasonal decomposition procedure:

Q2Q3Q4Q1

Comparison with seasonal indices obtained Comparison with seasonal indices obtained by ratioby ratio--toto--movingmoving--average method*average method*

122.5Q4

81.480.898.92Q373.974.491.15Q2

71.571.387.39Q1

From logged regression

Rescaled to Q4=100

Seasonal indexQuarter

* using Time Series/Seasonal Decomposition procedure on NETSALES

Almost identical!Almost identical!

Just for fun, let’s try a Just for fun, let’s try a quadratic quadratic trend line with the 3 seasonal trend line with the 3 seasonal dummies. RMSE and MAPE are $191M and 4%, much better dummies. RMSE and MAPE are $191M and 4%, much better than the linear trend model and not much worse than the than the linear trend model and not much worse than the trendtrend--change model. However, this sort of “polynomial curvechange model. However, this sort of “polynomial curve--fitting” is not a recommended method of predicting the future.fitting” is not a recommended method of predicting the future.

Time Sequence Plot for NETSALESQuadratic trend = -1.186E8 + 1.109E6 t + -2505. t^2 + 3 regressors

Q2/97 Q2/00 Q2/03 Q2/06 Q2/09 Q2/1213

23

33

43

53(X 1.E5)

NET

SALE

S

A 3A 3--year extrapolation of a downward quadratic curve is not year extrapolation of a downward quadratic curve is not very credible!very credible!

TakeTake--awaysawaysThis analysis has illustrated how regression can be This analysis has illustrated how regression can be used for the estimation of seasonal patterns via the used for the estimation of seasonal patterns via the use of dummy variables, and changes in trend via use of dummy variables, and changes in trend via “ramp” variables, as well as for curve“ramp” variables, as well as for curve--fitting.fitting.

Either additive or multiplicative seasonal indices can Either additive or multiplicative seasonal indices can be estimated within a regression model, depending be estimated within a regression model, depending on whether or not a log transformation is used.on whether or not a log transformation is used.

CurveCurve--fitting is usually fitting is usually notnot a good way to forecast a good way to forecast outside the sample!outside the sample!

Before using any model to forecast into the future, Before using any model to forecast into the future, make sure its fit to the make sure its fit to the recent recent past is good, and make past is good, and make sure you believe that its assumptions will continue to sure you believe that its assumptions will continue to hold in the future.hold in the future.

Another approach: seasonal lagsAnother approach: seasonal lagsAs an alternative to seasonal adjustment or As an alternative to seasonal adjustment or dummy variables, seasonality can be captured dummy variables, seasonality can be captured by using a by using a seasonal lagseasonal lag, i.e., the one, i.e., the one--yearyear--prior prior value of the dependent variable, as a value of the dependent variable, as a regressorregressor..

If s is the number of periods in a season (e.g., If s is the number of periods in a season (e.g., s=4 for quarterly data), it is often worth trying s=4 for quarterly data), it is often worth trying lags 1, s, and s+1.lags 1, s, and s+1.

In this approach, there is no explicit estimation of In this approach, there is no explicit estimation of seasonal indices. Instead, last year’s seasonal seasonal indices. Instead, last year’s seasonal pattern is used as a model for predicting the pattern is used as a model for predicting the future seasonal pattern. future seasonal pattern.

Here is a mean model with lags 1, 4, and 5 of sales used as Here is a mean model with lags 1, 4, and 5 of sales used as independent variables. The model type has been set to “ARIMA” independent variables. The model type has been set to “ARIMA” with zeroes in all the input fields (i.e., no differencing or ARwith zeroes in all the input fields (i.e., no differencing or AR/MA /MA factors), which is exactly equivalent to a mean model, except thfactors), which is exactly equivalent to a mean model, except that at it allows us to suppress the otherwise automatic it allows us to suppress the otherwise automatic backforecastingbackforecastingof missing values of lagged variables.of missing values of lagged variables.

Not bad! RMSE=186, MAPE=3.6%, decentNot bad! RMSE=186, MAPE=3.6%, decent--looking looking residual time series & autocorrelation plots, even with residual time series & autocorrelation plots, even with no special treatment for 2001.no special treatment for 2001.

This model bases the forecast entirely on what has This model bases the forecast entirely on what has happened in the last 5 quarters. happened in the last 5 quarters.

Time Sequence Plot for NETSALESARIMA(0,0,0) with constant + 3 regressors

Q2/97 Q2/00 Q2/03 Q2/06 Q2/0913

23

33

43

53(X 1.E5)

NE

TSA

LES

Updated tournament Updated tournament results: model with lagged results: model with lagged variables is Model D.variables is Model D.

The coefficient of lag(NETSALES,4) is almost exactly equal to The coefficient of lag(NETSALES,4) is almost exactly equal to 1, the coefficients of lag(NETSALES,1) and lag(NETSALES,5) 1, the coefficients of lag(NETSALES,1) and lag(NETSALES,5) are opposite in sign and roughly equal in magnitude (are opposite in sign and roughly equal in magnitude (±±0.7), and 0.7), and the constant is the constant is ≈≈ 430. The forecasting equation is therefore:430. The forecasting equation is therefore:

In words, the predicted increase over the In words, the predicted increase over the same quarter last same quarter last yearyear is equal to 430 plus 0.7 times the is equal to 430 plus 0.7 times the previous previous quarterquarter’’s s increase over the same quarter lastincrease over the same quarter lastyear, thus it predicts a general upwardyear, thus it predicts a general upwardseasonseason--toto--season season change,butchange,but adapts adapts to recent sameto recent same--quarter results.quarter results.

4 1 5ˆ 430 0.7( )t t t ty y y y− − −= + + −

How does it work, and why does it make sense? We’ll look How does it work, and why does it make sense? We’ll look deeper into models like this when we get to the ARIMA part of deeper into models like this when we get to the ARIMA part of the course, but here’s the basic logic of this model:the course, but here’s the basic logic of this model:

Effects of pricing and promotionsEffects of pricing and promotions

Here the variables are pounds sold of two sizes of bags of chipsHere the variables are pounds sold of two sizes of bags of chips(XL and XXL), as well as price(XL and XXL), as well as price--perper--bag and poundsbag and pounds--onon--display. display. (104 weekly observations at a regional supermarket chain)(104 weekly observations at a regional supermarket chain)

INDEX

LBXL

LBXXL

PRXL

PRXXL

DISPXL

DISPXXL

ScatterplotScatterplot matrixmatrix

Significant negative price effect for the XL sizeSignificant negative price effect for the XL size——but is it but is it linear? Also, a positive display effect. (3 outliers at top linear? Also, a positive display effect. (3 outliers at top of display of display scatterplotscatterplot correspond to very low prices.)correspond to very low prices.)

Correlation matrixCorrelation matrix

Significant Significant correlations with correlations with both price and both price and displaydisplay

AutoAuto-- and crossand cross--correlation correlation plots for LBXL: possibly a plots for LBXL: possibly a significant autosignificant auto--correlation correlation at lag 1… at lag 1…

The most significant crossThe most significant cross--correlation is at lag correlation is at lag --1, but this 1, but this is not useful for predicting is not useful for predicting LBXL from DISPXL (instead LBXL from DISPXL (instead the reverse!)the reverse!)

……and a crossand a cross--correlation correlation with DISPXL at lag 1 as well with DISPXL at lag 1 as well as lag 0? as lag 0?

Not usefulNot useful

Regression with all “likely suspects”, Regression with all “likely suspects”, including lagged variablesincluding lagged variables

Lagged variables turn Lagged variables turn out not to be significantout not to be significant

Tools for selecting Tools for selecting regressorsregressorsAfter a model has been fitted, After a model has been fitted, tt--statistics of statistics of coefficients (and their coefficients (and their pp--values) indicate whether values) indicate whether some variables can be removed.some variables can be removed.

Rule of thumb: Rule of thumb: tt less than 2 in magnitude (less than 2 in magnitude (p>p>.05) .05) suggests variable can be removed.suggests variable can be removed.

It’s not It’s not requiredrequired to remove a marginal variable if it to remove a marginal variable if it is strongly supported by intuition, but beware of is strongly supported by intuition, but beware of including including severalseveral marginal variables at once.marginal variables at once.

After manually removing the After manually removing the insignificantinsignificant lagged variables…lagged variables…

DsplayDsplay variables now have variables now have dubious significance when dubious significance when both are included in the both are included in the model model

Automatic stepwise selectionAutomatic stepwise selectionStepwise regression (forward and backward) is Stepwise regression (forward and backward) is available as a model option.available as a model option.

OK when used to automate what you would have OK when used to automate what you would have done anyway by hand, but done anyway by hand, but use with careuse with care..

Backward Backward stepwise automates the process of stepwise automates the process of sequentially removing the least significant variable sequentially removing the least significant variable until only significant variables remainuntil only significant variables remain

ForwardForward stepwise automates the process of stepwise automates the process of sequentially adding the variable that sequentially adding the variable that would bewould be most most significant upon being the next one enteredsignificant upon being the next one entered

Automatic stepwise selectionAutomatic stepwise selection

This is a rightThis is a right--mousemouse--button Analysis Option button Analysis Option in the Multiple in the Multiple Regression procedureRegression procedure

Automatic stepwise selectionAutomatic stepwise selection

FF--toto--enter and enter and FF--toto--remove determine the remove determine the tt--stats stats needed for a variable to enter or stay in model.needed for a variable to enter or stay in model.

FF = = tt--squared, hence squared, hence FF=4 corresponds to =4 corresponds to tt=2.=2.

Resist the urge to lower the threshold in order to Resist the urge to lower the threshold in order to “find” or “keep” more variables“find” or “keep” more variables——danger of danger of overfittingoverfitting!!

Automatic methods, continuedAutomatic methods, continuedAutomatic modelAutomatic model--fitting techniques aren’t a substitute fitting techniques aren’t a substitute for your own good judgment.for your own good judgment.

They are only as good (or bad) as the set of variables They are only as good (or bad) as the set of variables you give them to work with.you give them to work with.

They won’t find omitted variables or suggest They won’t find omitted variables or suggest transformations of variablestransformations of variables

Dangerous for “data snooping” in large, uncriticallyDangerous for “data snooping” in large, uncritically--chosen sets of variables: outchosen sets of variables: out--ofof--sample validation sample validation becomes becomes veryvery crucial crucial

At the end of the day, At the end of the day, youyou (not the computer) are (not the computer) are responsible for the model!responsible for the model!

Backwards stepwise eliminates one Backwards stepwise eliminates one more variable…more variable…

After removing DISPXL (the After removing DISPXL (the least significant variable in the least significant variable in the 44--variable model), DISPXXL variable model), DISPXXL rises just above the t=2 rises just above the t=2 threshold of significance.threshold of significance.

Normal Probability Plot

-1000 -500 0 500 1000 1500

RESIDUALS

0.115

2050809599

99.9

perc

enta

ge

ResidualResidual--vsvs--predicted predicted and residual probability and residual probability plot are suggestive of a plot are suggestive of a nonlinear relationship nonlinear relationship and/or nonand/or non--normal errorsnormal errors

INDEX

LOG(LBXL)

LOG(LBXXL)

LOG(PRXL)

LOG(PRXXL)

DISPXL

DISPXXL

Try logging both poundsTry logging both pounds--sold and pricesold and price

Note: we Note: we cancan’’ttdirectly log the directly log the display variables display variables since they since they contain zeroescontain zeroes

The poundsThe pounds--sold vs. price sold vs. price relationship relationship now appears now appears more linearmore linear

Correlations Correlations are similar to are similar to

those those obtained obtained earlier…earlier…

AutoAuto-- and crossand cross--correlation correlation plots are also similar. plots are also similar. However, the time series However, the time series plot now looks much more plot now looks much more ““normalnormal”” (less spiky)(less spiky)

Regression resultsRegression results

Coefficients of logged variables can now Coefficients of logged variables can now be interpreted as be interpreted as elasticitieselasticities:: a 1% a 1% increase in PRXL yields a 1.8% increase in PRXL yields a 1.8% decrease in LBXLdecrease in LBXL

Interestingly, the crossInterestingly, the cross--elasticity elasticity of PRXXL with LBXL is almost of PRXXL with LBXL is almost exactly opposite (+1.88), as if the exactly opposite (+1.88), as if the products are substitutesproducts are substitutes

Normal Probability Plot

-0.43 -0.23 -0.03 0.17 0.37

LRESIDUALS

0.115

2050809599

99.9pe

rcen

tage

ResidualResidual--versusversus--predicted and residual predicted and residual probability plot now probability plot now look better!look better!

NotNot--logged model refitted in the logged model refitted in the Forecasting procedureForecasting procedure

Logged model refitted in the Logged model refitted in the Forecasting procedureForecasting procedure

HeadHead--toto--head head model model comparisons comparisons in original in original unitsunits

Here model C is Here model C is obtained by obtained by applying an AR(1) applying an AR(1) autocorrelation autocorrelation correction to model correction to model B (a very slight B (a very slight improvement)improvement)

With 20 points held out for validationWith 20 points held out for validation

Error statistics of all Error statistics of all three models are three models are significantly higher in the significantly higher in the validation period. Have validation period. Have we we overfittedoverfitted the data, or the data, or are the last 20 points are the last 20 points exceptional?exceptional?

With 50 points held out for validationWith 50 points held out for validation

Now the error Now the error statistics are quite statistics are quite similar. Evidently the similar. Evidently the models are not models are not overfittedoverfitted..

Coefficient estimates for model with Coefficient estimates for model with 50 points held out50 points held out

Price coefficients are almost the Price coefficients are almost the same as before (good!). The same as before (good!). The display coefficients are also in display coefficients are also in the same ballpark as before. the same ballpark as before. They are no longer They are no longer technicallytechnicallysignificant, but this is because significant, but this is because the standard errors are larger the standard errors are larger when a smaller sample is used to when a smaller sample is used to estimate them.estimate them.

Class 6 recapClass 6 recapFitting regression models to time series data Fitting regression models to time series data

Economic interpretation of coefficientsEconomic interpretation of coefficients

How to model seasonality with regressionHow to model seasonality with regression

LogLog--log (constant elasticity) modelslog (constant elasticity) models

Stepwise variable selectionStepwise variable selection