bdx 2016 - kevin lyons & yakir buskilla @ exelate
TRANSCRIPT
![Page 1: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/1.jpg)
Online LearningThe Future of Audience Segmentation is Here
Kevin Lyons + Yakir Buskilla
![Page 2: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/2.jpg)
Models that build profitable marketing audiences at scale...
Finding more of your best customers: High-income business professional
![Page 3: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/3.jpg)
The Modeling Process, simplified
![Page 4: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/4.jpg)
2012 2015
30 - 40 modelslevering billions of events
Creating 100 million + scores
over 1000 models‘leveraging’ trillions of events
Creating 150 billion+ scores / day
The Challenge
![Page 5: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/5.jpg)
In other words, we simply need ….
![Page 6: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/6.jpg)
A system creates as many models as we want, when
we want them, that dynamically adapts in real-time
to changing conditions
○ Automatically creates, validates, ships, and
monitors models, with a capacity that scales
to 10s of thousands of models
The Opportunity
What we really need:
![Page 7: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/7.jpg)
Online models evolve & adapt over time, in
reaction to a changing environment with each
and every event
Given a complete data set, a batch
model is created in entirety all at once
Introducing Online Learning
Batch Online Learning
Creation Evolution
![Page 8: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/8.jpg)
large-scale data storage
large-scaledata schelping
painful data aggregation
lots of manual everything
Harder to build models, but easier to evaluate
limited data storage, mostly for monitoring
event-leveldata streams
light data aggregation
lots of automatic everything
Easier to build, but harder to evaluate (& support)
Batch Models (Offline) vs. Online Learning
Online LearningBatch Models (Offline)
![Page 9: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/9.jpg)
● Outperformed both L2 and Elastic Net
● Leverages small (‘micro’) batches
● Validates and monitors models in real time
● Alerts team when models are not behaving
Some Techno Mumbo Jumbo
Stochastic gradient descent with L1 regularization
![Page 10: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/10.jpg)
eXelate.com @eXelate
Technical Solutions
How do we do it?
![Page 11: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/11.jpg)
eXpresso Serving Cluster
10B events/day
260 nodes across4 data centers
eXtream Modeling Cluster
160B models/day
85 nodes across4 data centers
JGroups
DistributedMessaging
Serving Layer
![Page 12: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/12.jpg)
Online LearningBatch Models (Offline)
Batch
Predefined ratio
Predefined feature selection
One time Validation
Streaming
Downsampling
Automated feature selection
Ongoing data cleaning
Ongoing validation
The Online Learning Challenge
![Page 13: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/13.jpg)
● All necessary data already exists in eXtream
● The cluster’s processing resources can be better utilized
● eXtream addresses most performance / scalability requirements
● Scoring mechanism already exists
eXtream as a Framework for Online Learning
Why it works...
![Page 14: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/14.jpg)
Online Learning Flow
![Page 15: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/15.jpg)
● Labeling Mechanism - customer defined target audience
Events Classification
![Page 16: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/16.jpg)
● Downsampling mechanism● Burst tolerance● Duplicate entries
Dataset Preparation
![Page 17: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/17.jpg)
● Blacklist● Whitelist● Automatic Tuning
Features Selection
![Page 18: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/18.jpg)
● Sliding window of recent events● 60/40 not-converted/converted ratio● Various accuracy metrics (lift, precision, recall, confusion matrix)● Decide if the model is ready for making predictions
Model Validation
![Page 19: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/19.jpg)
● Two phases (Scoring, Re-code)● Scale vs Accuracy tradeoff
Predictions Mechanism
![Page 20: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/20.jpg)
Scalability / Performance
Thousands of
Concurrent Models: High Throughput:
billions of training events per daytraining, validation, scoring
![Page 21: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/21.jpg)
Why do we need it?● Store the models in one common place
● Persistency
● Built-in replication
● Aerospike has built in limitation for object size - 1MB
○ Developed sharding mechanism for storing models on Aerospike
Scalability / Performance
Why do we need it?
Large object issue on Aerospike
![Page 22: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/22.jpg)
The solution is Aerospike fast built-in replication
Cross Data Center Learning
● Low Volume Models
● Traffic Redirection
![Page 23: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/23.jpg)
Monitoring- Why do we need it?
thousands of models
automatically created by users
some models won’t converge
![Page 24: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/24.jpg)
Monitoring- Real Time
![Page 25: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/25.jpg)
Monitoring- Aggregation
![Page 26: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/26.jpg)
Monitoring- DS Bot
![Page 27: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/27.jpg)
eXelate.com @eXelate
Case study
Working in action
![Page 28: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/28.jpg)
● The ideal candidate for digital media expands and even subtly shifts in real time● Real-time modeling tracks and reacts to these changes as they happen, with 2x CPA
improvement over a batch model
The Times, They Are A-Changin’
Market: Downgrading a country’s credit ratings
● Holiday shopping is very different from the rest of the year, particularly Cyber Monday ● AM changes in Eastern US are applied to the Pacific coast before the madness begins
Audiences: Cyber Monday frenzies
● … after the campaign starts, effecting the ideal audience● No need to panic; modeled audience automagically adjust
Product: A product offering is revised
![Page 29: BDX 2016 - Kevin lyons & yakir buskilla @ eXelate](https://reader031.vdocuments.mx/reader031/viewer/2022030316/58751a7b1a28ab05598b74d1/html5/thumbnails/29.jpg)
Scores of self-maintaining models that constantly adapt to our ever changing conditions
Happiness Renewed...