presentation title goes here · example data •in this framework forecast horizon becomes another...

Session Goals

Motivation

Results Preview

Background

Tuning Models: Why and How

Primer on K-Fold Cross-Validation (K=4)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Fold 1

Fold 2

Fold 3

Fold 4Test DataValidation Fold 4

Validation Fold 3

Validation Fold 2Training Fold 2

Training Fold 3 Training Fold 3

Training Fold 4

Training Fold 2 Test Data

Test Data

Entire Dataset

Test DataValidation Fold 1Training Fold 1

Cross-Validation for Time SeriesApr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Not Used

Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Test DataNot Used

Entire Dataset

Fold 3Test Data

Fold 4Test Data

Training Fold 1

Training Fold 2

Training Fold 3

Training Fold 4

Validation Fold 1

Validation Fold 2

Validation Fold 3

Validation Fold 4

Not Used Test DataFold 1

Fold 2

Application Details

Using Out-of-Fold Forecasts as Features

Results• Mean absolute error:

• 𝑚𝑎𝑒 = Τσ𝑖=1𝐼 𝑦𝑖 − 𝑓𝑖 𝐼

• Displayed are improvements relative to mean forecast

• Mean is average of seasonal naïve, ARIMA, ETS, Elastic Net, and k-NN

• Ensemble: forecast from regression model with individual forecasts are features and revenue is target

• Numbers displayed are % differences between methods

• E.g. for Device 1, Country 1:𝑚𝑎𝑒𝐴𝑅𝐼𝑀𝐴

𝑚𝑎𝑒𝑀𝑒𝑎𝑛− 1 = −8.43%

Device Country S.Naive ARIMA ETS Elastic Net k-NN Ensemble

1 1 -2.28 -8.43 100.46 -16.89 -16.76 -82.45

2 -6.70 -6.52 100.47 -34.30 26.71 -70.30

3 -14.68 -32.72 -32.30 332.31 -18.18 -78.87

4 -27.11 -14.78 116.93 24.98 5.52 -74.18

5 -36.25 -25.23 -15.59 143.15 8.45 -74.84

6 -0.42 -8.48 7.92 104.55 -2.36 -29.49

7 -1.92 13.86 80.00 -15.92 69.85 -46.26

8 18.55 10.84 160.08 175.02 63.96 -50.27

Average -8.85 -8.93 64.75 89.11 17.15 -63.33

2 1 12.25 -0.94 132.48 -44.36 -32.14 -91.92

2 26.89 37.97 -20.90 57.00 41.67 -74.33

3 25.08 5.59 153.18 -47.53 -57.76 -80.30

4 38.15 203.74 227.58 3.70 -45.91 -71.69

5 32.80 178.00 90.10 140.05 -23.77 -53.79

6 36.17 100.83 170.08 -15.40 -37.35 -73.77

7 27.36 50.57 160.55 -82.08 -82.59 -89.11

Average 28.39 82.25 130.44 1.63 -33.98 -76.42

Takeaways

Example Data• In this framework forecast horizon

becomes another feature

• Other features include year, quarter, and month, perhaps trend

• Must include lags based on horizon to avoid peeking into the future

• Can use rolling forecasts from univariate time series models, e.g. ETS or ARIMA, as features

• “Driver” time series can be handled similarly

• e.g. Bing query data

Index 2016-M01 2016-M02 2016-M03 2016-M04 2016-M05 2016-M06

Revenue Y1 Y2 Y3 Y4 Y5 Y6

Month Horizon Revenue Lag 1 Lag 2 ETS Fcst

2016-M04 1 Y4 Y3 Y2 F_4|3

2016-M04 2 Y4 Y2 Y1 F_4|2

2016-M04 3 Y4 Y1 Y0 F_4|1

2016-M05 1 Y5 Y4 Y3 F_5|4

2016-M05 2 Y5 Y3 Y2 F_5|3

2016-M05 3 Y5 Y2 Y1 F_5|2

2016-M06 1 Y6 Y5 Y4 F_6|5

2016-M06 2 Y6 Y4 Y3 F_6|4

2016-M06 3 Y6 Y3 Y2 F_6|3

Cross-Validation with HorizonsApr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Apr2016 May2016 Jun2016 Jul2016 Aug2016 Sep2016 Oct2016 Nov2016 Dec2016 Jan2017 Feb2017 Mar2017 Apr2017 May2017 Jun2017

Horizon 1 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

Horizon 2 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

Horizon 3 Tr Tr Tr V1 V2 V3 V4 V5 V6 V7 V8 V9 V10

Tr Training V1 Validation, real quarter V2 Validation, pseudo-quarter

Entire Dataset