Counting what will Counting what will countcount
Does your dashboard Does your dashboard predict ?predict ?
Koen PauwelsKoen Pauwels
Amit JoshiAmit Joshi
Marketing dashboardsMarketing dashboards
• Marketing accountability & accelerating Marketing accountability & accelerating changechange
• Limits to human processing capacity still there: Limits to human processing capacity still there: MSIMSI: ‘separate signal from noise’, ‘dashboards’: ‘separate signal from noise’, ‘dashboards’
• Top management interest: ‘substantial effort’ Top management interest: ‘substantial effort’ by 40% of US and UK companies (Clark ea by 40% of US and UK companies (Clark ea 2006)2006)
• But current applications often fail to impress:But current applications often fail to impress:
Marketing Scientists can help raise the bar !Marketing Scientists can help raise the bar !
Academic research focusAcademic research focus
Reibstein ea (2005) discuss 5 development Reibstein ea (2005) discuss 5 development stages:stages:
1)1) Identify key metrics (align with firm goals)Identify key metrics (align with firm goals)2)2) Populate the dashboard with dataPopulate the dashboard with data3)3) Establish relation between dashboard itemsEstablish relation between dashboard items4)4) Forecasting and ‘what if’ analysisForecasting and ‘what if’ analysis5)5) Connect to financial consequencesConnect to financial consequencesMost dashboards yet to move beyond stage 2 !Most dashboards yet to move beyond stage 2 !Need research to select best metrics, relate to Need research to select best metrics, relate to
performanceperformance
You’ve got 33 x 3 = 99 You’ve got 33 x 3 = 99 variablesvariables
• Market variablesMarket variables: price national brand, store brand, : price national brand, store brand, ∆∆• AwarenessAwareness: top-of-mind, aided, unaided, ad awareness: top-of-mind, aided, unaided, ad awareness• Trial/usageTrial/usage: ever, last week, last 4 weeks, 3 months : ever, last week, last 4 weeks, 3 months • Liking/Satisfaction: Liking/Satisfaction: given aware and given triedgiven aware and given tried• Preference:Preference: favorite brand, will it satisfy future needs? favorite brand, will it satisfy future needs?• Purchase intentPurchase intent: given aware and given tried: given aware and given tried• Attribute ratingsAttribute ratings: taste, quality, trust, value, fun, feel: taste, quality, trust, value, fun, feel• Usage occasionUsage occasion: home, on the go, afternoon, entertain: home, on the go, afternoon, entertainNeed to reduce 99 to 6-10 metrics (US) or 10-20 Need to reduce 99 to 6-10 metrics (US) or 10-20
(UK)(UK)
Metric deletion rules Metric deletion rules (Ambler 2003)(Ambler 2003)
1)1) Does the metric rarely change?Does the metric rarely change?
2)2) Is the metric too volatile to be reliable? Is the metric too volatile to be reliable? Univariate tests on time series propertiesUnivariate tests on time series properties
3)3) Is metric leading indicator of market Is metric leading indicator of market outcome? outcome? Pairwise tests of metric with Pairwise tests of metric with performanceperformance
4)4) Does the metric add sufficient explanatory Does the metric add sufficient explanatory power to existing metrics? power to existing metrics? Econometric models to explain Econometric models to explain performanceperformance
This researchThis research• Univariate: st. dev., coef. of variation, evolution Univariate: st. dev., coef. of variation, evolution • Pairwise: Pairwise: Granger CausalityGranger Causality test with test with
performanceperformance• Explanatory powerExplanatory power: regression model : regression model
comparisoncomparison1)1) Stepwise regression Stepwise regression (Hocking 1976, Meiri ea 2005)(Hocking 1976, Meiri ea 2005)
2)2) Reduced Rank Regression Reduced Rank Regression (Reinsel and Velu 1998)(Reinsel and Velu 1998)
3)3) Forecast error variance decomposition, based Forecast error variance decomposition, based on Vector Autoregressive Model on Vector Autoregressive Model (Hanssens 1998)(Hanssens 1998)
• AssessmentAssessment: forecasting accuracy hold-out : forecasting accuracy hold-out samplesample
• Managerial controlManagerial control: impact size and lead time: impact size and lead time
Stepwise regressionStepwise regression
• Automatic selection based on statistical criteriaAutomatic selection based on statistical criteria
Objective: select set of metrics with highest RObjective: select set of metrics with highest R22
• Forward: add variables with lowest p-valueForward: add variables with lowest p-value
Backward: delete variables with highest p-valueBackward: delete variables with highest p-value• Unidirectional: considers one variable at a timeUnidirectional: considers one variable at a time
Stepwise: checks all included against criterionStepwise: checks all included against criterion
Combinatorial: evaluates every combinationCombinatorial: evaluates every combination
Reduced Rank Reduced Rank RegressionRegression
• Uses correlation of key metrics and Uses correlation of key metrics and performanceperformance
Yi = Xi’C + εi with Yi (m x 1) and Xi (n x 1)Yi = Xi’C + εi with Yi (m x 1) and Xi (n x 1)
C (m x n) has C (m x n) has rank r ≤ min (m, n)rank r ≤ min (m, n)
Restriction: m – r linear restrictions on CRestriction: m – r linear restrictions on C
Maximize explained variance under Maximize explained variance under restrictionrestriction
• Originally shrinkage regression (Aldrin Originally shrinkage regression (Aldrin 2002), now for selecting best combination 2002), now for selecting best combination variablesvariables
Forecast Forecast variance variance decompositiondecomposition
• Based on Vector Autoregressive ModelBased on Vector Autoregressive Model
A ‘dynamic RA ‘dynamic R22’, FEVD calculates the ’, FEVD calculates the percentage of variation in performance that percentage of variation in performance that can can be attributed to changes in each of can can be attributed to changes in each of the endogenous variables (Hanssens 1998, the endogenous variables (Hanssens 1998, Nijs ea 2006)Nijs ea 2006)
• Measures the relative performance impact Measures the relative performance impact over time of shocks initiated by each over time of shocks initiated by each endogenous varendogenous var
• We consider the FEVD at 26 weeksWe consider the FEVD at 26 weeks
Methods share 4/10 Methods share 4/10 metricsmetrics
StepwiseStepwise Reduced Reduced RR
FEVDFEVD
MarketMarket pricenb, -pricenb, -stst
pricenb, -pricenb, -stst
pricenb, -pricenb, -stst
AwarenesAwarenesss
awareunnawareunnbb
awunnb, -awunnb, -stst
awareunnawareunnbb
Usage/Usage/trialtrial
tried3mtried3m∆∆ tried3mtried3m∆∆ tried4wnbtried4wnb
Purch. Purch. IntentIntent
piawarenpiawarenbb
AffectAffect satistriedsatistriednbnb
satistriedsatistriednbnb
liketriednliketriednbb
Attribute Attribute RatingsRatings
satisnb, satisnb, feelstfeelst
funfun∆∆, , taste taste ∆∆, , trust trust ∆∆
satisnb, satisnb, trusttrust∆, ∆, qualqual∆∆
Usage Usage occasionoccasion
afternoonafternoonnb nb entertainentertainnbnb
entertainentertainnbnb
afternoonafternoonnb nb entertainentertainnbnb
Stepwise scores within Stepwise scores within samplesample
0
10
20
30
40
50
60
70
80
90
100
R-SQUARED ADJUSTED R-SQUARED
STEPWISE REGRESSION REDUCED RANK REGRESSION FORECAST ERROR VARIANCE DECOMPOSITION
But sucks out-of-sampleBut sucks out-of-sample
0
5
10
15
20
25
MEAN AVERAGE PERCENTAGE ERROR THEIL'S INEQUALITY COEFFICIENT
STEPWISE REGRESSION REDUCED RANK REGRESSION FORECAST ERROR VARIANCE DECOMPOSITION
Sales Impact Size and Sales Impact Size and Timing Timing
1% change 1% change inin
Short-termShort-term Long-termLong-term Wear-inWear-in Wear-outWear-out
PRICENBPRICENB -161,794-161,794 -84,417-84,417 00 88
PRICESTPRICEST 71,56171,561 121,997121,997 00 11
AFTERNOONAFTERNOON 32, 12932, 129 32, 12932, 129 00 00
TRUST∆TRUST∆ 23,70723,707 23,70723,707 00 00
LIKEgiventriednbLIKEgiventriednb 00 40,42040,420 11 00
QUALITY∆QUALITY∆ 00 66,96366,963 11 00
TRIED4WNBTRIED4WNB 00 72, 48172, 481 22 44
SatisfyingNBSatisfyingNB 00 46,78846,788 22 00
EntertainfriendsNBEntertainfriendsNB 00 60,07160,071 22 33
AWAREUNNBAWAREUNNB 00 68,53768,537 33 44
Value (price-quality) matters Value (price-quality) matters right now, Awareness and right now, Awareness and
Trial soon ! Trial soon !
0
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
0 1 2 3 4
Weeks till peak impact
Lo
ng
-term
Sale
s I
mp
act PRIC
EQUALITY
Satisfying
Entertain friends
Tried last month
Unaided Awareness
TRUST
Afternoon Lift
Like if tried
Summary dashboard Summary dashboard intuitionintuition
1) To increases sales immediately (0-1 1) To increases sales immediately (0-1 weeks)weeks)
a) promote on price and on afternoon lift usagea) promote on price and on afternoon lift usage
b) communication focus on quality, affect, trustb) communication focus on quality, affect, trust
2) To increase sales soon (2-3 weeks)2) To increase sales soon (2-3 weeks)
a) provide free samples (to up ‘tried last a) provide free samples (to up ‘tried last month’)month’)
b) focus on satisfying and entertainment useb) focus on satisfying and entertainment use
c) advertise for unaided awarenessc) advertise for unaided awareness
Conclusion: P-model for Conclusion: P-model for dashboardsdashboards
1.1. Which metrics are leading indicators?Which metrics are leading indicators?
Granger causality testsGranger causality tests
2.2. Explain most of performance dynamicsExplain most of performance dynamics
Forecast error Variance decomposition Forecast error Variance decomposition
3.3. Forecast multivariate baseline with Vector Forecast multivariate baseline with Vector Autoregressive or Error Correction modelAutoregressive or Error Correction model
4.4. Displays Displays timingtiming and and size size of sales impactof sales impact
Your Questions ?Your Questions ?