exploiting full potential of predictive analytics on small ...exploiting full potential of...
TRANSCRIPT
![Page 1: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/1.jpg)
Exploiting full potential of predictive analytics on small data to drive business
outcomes
Adrian Foltyn, External Data Science Expert
TM Forum, Nice
15 May 2018
![Page 2: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/2.jpg)
Perfectly Portioned Ingredients For 3-5 Meals
Per Week
Personalised Fresh Food, Locally
Sourced
Easily Managed Via Subscription
Platform
1 Box Delivered Weekly To The
Door
NoPlanning
NoShopping
NoWaste
HelloFresh breaks the dinner routine by continuously innovating both service and product
![Page 3: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/3.jpg)
Disrupting the supply chain by cutting middlemen, ensuring higher margins and fresher products
![Page 4: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/4.jpg)
5
HelloFresh global footprint
![Page 5: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/5.jpg)
How we use data science / machine learning
DATA SCIENCE @ HF
Fraud detection
Marketing attribution
Lifetime / churn prediction
Recommendation engines
Demand forecasting
minimize cost maximize revenue
Generalized Additive Models
Support Vector Regressions
Random Forests
Extreme Gradient Boosting
Bayesian networks
Collaborative filtering
Deep learning CNNs
ARIMA & other time series
models
Hidden Markov Models
Graph databases
![Page 6: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/6.jpg)
Myself: a drift between consulting and data science
▪ Quant methods and computational psychoacoustics
▪ Demand forecasting
▪ Market research & business intelligence
▪ Data Science in strategic consulting
▪ Data Science in-house
![Page 7: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/7.jpg)
Why bother about small data? Isn’t it dead by now?
▪ Not all areas of business produce big data
▪ Several business functions use / produce both small and big data, e.g. :
‒ Demand planning / forecasting
‒ Attribution & ROI of marketing activities
▪ As a result, top-down and bottom-up models appear
‒ Top-down focuses on trends of phenomena, predictors, aggregated values
‒ Bottom-up drills down to individual customer / transaction etc.
▪ Both model types require validation and could provide it for each other….
![Page 8: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/8.jpg)
Demand forecasting challenges in a subscription meal kit business
▪ Huge cost impact
▪ Short time series
▪ High variance of pausing, strong seasonal (holiday) effects
▪ New conversions highly dependent on marketing plans
▪ Certain predictors difficult to track (e.g. errors in delivery)
▪ Large set of numbers to be delivered
‒ Splits into box types, recipes, delivery day, delivery time, …
▪ Tech legacy
What can we cook from these ingredients ? ☺
![Page 9: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/9.jpg)
First decision: forecast top-down or bottom-up ?
CustomerID
Weeks from activation
Weeks from last pause
Weeksfrom last meal swap
No. of meal swaps total
No. ofboxes in total
Box type
……. Probabilityof getting a box
….. ….. ….. ….. ….. ….. ….. ….. 0.4
….. ….. ….. ….. ….. ….. ….. ….. 0.7
….. ….. ….. ….. ….. ….. ….. ….. 0.5
….. ….. ….. ….. ….. ….. ….. ….. 0.6
Total 0.55
Sales (boxes)**
Outlook of actives
Outlook of pauses
………..
`
** Dummy data in all charts
~
+
+
![Page 10: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/10.jpg)
Why do we need a top-down forecasting model?
Should I stay (or should I
go)?
Shall I take a break?
Do I care to see my options
?
Do I swap my
meals?
CANCEL? PAUSE?TRUST DEFAULT MEAL CHOICE ?
SWAP MEALS?
Y
N
Y
Y
Y
N
N
N
• Each decision increases variance of final output
• In a bottom-up model those variances could mitigate each other or could explode…
• Top-down model (aggregate number of boxes) is much more stable
![Page 11: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/11.jpg)
Methodological challenges of short time series forecasting
▪ Possible lack of time series effects ‒ no significant autocorrelation of sales
‒ still, burning need to control for trend in data / ‘baseline’
▪ Dynamic business growth introduces sales disruptions
▪ Lots of predictors are inter-correlated‒ particularly true in the case when some
outlooks (early trends) of activeness, pausing and cancelling are available
Forecasting – like cooking – is a mixture of art and science ☺
![Page 12: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/12.jpg)
To our surprise, often times we found lack of time-related effects in forecast target
** Dummy data in all charts
Standard time series models cannot be the only tool we use….
Boxes shipped in one of countries **
Autocorrelation of differenced time series Partial autocorrelation of differenced ts
`
` `
![Page 13: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/13.jpg)
We needed to introduce dummies for disruptions in sales time series
** Dummy data in all charts
` `
Histogram of weekly sales in country 1 ** Time series of weekly sales in country 2 **
Mixture of 2 distributions due to major change in marketing spend
Price cut effect = shift of entire sales
![Page 14: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/14.jpg)
Feature selection is limited by size of input data and requires regularization
▪ Lots of predictors are highly correlated, even after controlling for trend / baseline
▪ This calls for regularization in both feature selection process and then in the model
▪ We use Lasso in feature selection and cubic shrinkage in Generalized Additive Models
``
![Page 15: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/15.jpg)
Addressing non-linearity: Generalized Additive Models
• Introduced by Hastie and Tibsharani in 1990
• ’a step taken from GLM towards non-parametric models’
• Instead of estimating parameters, for each variable GAM estimates a function composed of smoothing elements
• The functions f1, . . . , fp can be natural splines, smoothing splines, local regressions, technically even polynomials (not used)
• Parametric terms and 2D-smoothers are also allowed
Future sales ie. no. of boxes to be delivered X weeks from now=
b0 + f1(marketing spend) + f2(planned pauses to date) + f3(planned cancellations to date) + f4(holiday effect) + weekday effect + …
![Page 16: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/16.jpg)
Ensemble approach and root cause analysisBacktest in country 1 Backtest in country 2
• Neither standard time series nor average ensemble forecast work• Best forecast method selected by progressive cross validation is better (final.forecast)• Frequent review based on backtesting and root-cause analysis is even better
` `
Mo
del
err
or
bas
ed o
n 1
6-w
eek
pro
gres
sive
cro
ss-v
alid
atio
n
Mo
del
err
or
bas
ed o
n 1
6-w
eek
pro
gres
sive
cro
ss-v
alid
atio
n
![Page 17: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/17.jpg)
Next step: predicting user-level demand with deep learning
CNNs
Factorization / Word2Vec
![Page 18: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/18.jpg)
Marketing attribution problem
• Marketing spend for a company within multiple years, typically measured daily or weekly
• Question: how many conversions / how much revenue / how much CLV can be attributed to activity in each channel?
![Page 19: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/19.jpg)
Marketing attribution challenges in most e-commerce businesses
▪ Huge cost impact
▪ Short time series
▪ Varying granularity of input data
▪ Very often marketing data stored by multiple people in obscure ways (sheets, docs, no standardization)
▪ Lack of full attribution models to cross check results
▪ Dealing with counter intuitive results…
Is there a silver / golden bullet? ☺
![Page 20: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/20.jpg)
Here we go again: top-down or bottom-up approach?
CustomerID
Touchpoint Paid-Social
Touchpt.Affiliates
Touchpt. Bloggers
Touchpt….
Likelihoodof outdoor exposure
Likelihoodof TVexposure
…….
Number of boxes overfirst year(CLV)
….. ….. ….. ….. ….. ….. ….. ….. 10.5
….. ….. ….. ….. ….. ….. ….. ….. 2.5
….. ….. ….. ….. ….. ….. ….. ….. 5.3
….. ….. ….. ….. ….. ….. ….. ….. 7.6
Total 9.2
Number of boxes from
newly acquired
customers
Activity in TV**
Activity in PaidSocial
**
………..
`
** Dummy data in all charts
~
+
+
![Page 21: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/21.jpg)
Balancing business insight and simulation / prediction power
• Typically, statistics used doesn’t align exactly to desired business outcomes
• There is usually an inverse relationship between how well the model predicts and how interpretable are its components
• In marketing attribution, forcing intuitive constraints (non-negative contribution of channels, convex shape of response = saturation etc.) often affect fit and predictive strength
• Hitting sweet spot requires an iterative process of refining the model against business assumptions and usability / actionability
![Page 22: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/22.jpg)
Simulator for marketing attribution & ROI purposes based on a PCA + Bayesian network + GAM model
![Page 23: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/23.jpg)
Conclusions: Do’s for small data
▪ Combine bottom-up and top-down
▪ Experiment if no history available
▪ Ensemble your models wisely
▪ Back-test and root-cause-analyze
▪ Factor in iterations with business and make it part of model building
▪ Keep calm and explain discrepancies….
Forecasting / simulating is the art of saying what will happen and then
explaining why it didn’t…
![Page 24: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/24.jpg)
We’re hiring at HelloFresh!
▪ Data Scientists
‒ Python, R, Spark, Scala, ML + computer vision / NLP / other deep learning experience
▪ Machine Learning Engineers
‒ Python, Hadoop, Spark, Kafka, ML productionizing expertise
▪ Data Engineers
‒ Python, Hadoop, Spark, Kafka, Airflow, ETL experience
https://www.hellofresh.com/careers/
![Page 25: Exploiting full potential of predictive analytics on small ...Exploiting full potential of predictive analytics on small data to drive business outcomes Adrian Foltyn, External Data](https://reader034.vdocuments.mx/reader034/viewer/2022042312/5edb09a809ac2c67fa68b55a/html5/thumbnails/25.jpg)
Thanks!Any Questions?