paris ml meetup
TRANSCRIPT
![Page 1: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/1.jpg)
![Page 2: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/2.jpg)
Machine Learning @ Netflix(and some lessons learned)
Yves Raimond (@moustaki)
Research/Engineering Manager
Search & Recommendations
Algorithm Engineering
![Page 3: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/3.jpg)
Netflix evolution
![Page 4: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/4.jpg)
Netflix scale● > 69M members
● > 50 countries
● > 1000 device types
● > 3B hours/month
● 36% of peak US downstream traffic
![Page 5: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/5.jpg)
Recommendations @ Netflix
● Goal: Help members find content
to watch and enjoy to maximize
satisfaction and retention
● Over 80% of what people watch
comes from our recommendations
● Top Picks, Because you Watched,
Trending Now, Row Ordering,
Evidence, Search, Search
Recommendations, Personalized
Genre Rows, ...
![Page 6: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/6.jpg)
▪ Regression (Linear, logistic, elastic net)
▪ SVD and other Matrix Factorizations
▪ Factorization Machines
▪ Restricted Boltzmann Machines
▪ Deep Neural Networks
▪ Markov Models and Graph Algorithms
▪ Clustering
▪ Latent Dirichlet Allocation
▪ Gradient Boosted Decision Trees/Random Forests
▪ Gaussian Processes
▪ …
Models & Algorithms
![Page 7: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/7.jpg)
Some lessons learned
![Page 8: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/8.jpg)
Build the offline experimentation framework first
![Page 9: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/9.jpg)
When tackling a new problem● What offline metrics can we compute that capture what online improvements we’
re actually trying to achieve?
● How should the input data to that evaluation be constructed (train, validation,
test)?
● How fast and easy is it to run a full cycle of offline experimentations?
○ Minimize time to first metric
● How replicable is the evaluation? How shareable are the results?
○ Provenance (see Dagobah)
○ Notebooks (see Jupyter, Zeppelin, Spark Notebook)
![Page 10: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/10.jpg)
When tackling an old problem● Same…
○ Were the metrics designed when first running experimentation in that space still appropriate now?
![Page 11: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/11.jpg)
Think about distribution from the outermost layers
![Page 12: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/12.jpg)
1. For each combination of hyper-parameter
(e.g. grid search, random search, gaussian processes…)
2. For each subset of the training data
a. Multi-core learning (e.g. HogWild)
b. Distributed learning (e.g. ADMM, distributed L-BFGS, …)
![Page 13: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/13.jpg)
When to use distributed learning?● The impact of communication overhead when building distributed ML
algorithms is non-trivial
● Is your data big enough that the distribution offsets the communication overhead?
![Page 14: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/14.jpg)
Example: Uncollapsed Gibbs sampler for LDA
(more details here)
![Page 15: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/15.jpg)
Design production code to be experimentation-friendly
![Page 16: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/16.jpg)
Idea Data
Offline Modeling
(R, Python, MATLAB, …)
Iterate
Implement in production
system (Java, C++, …)
Missing post-processing logic
Performance issues
Actual outputProduction environment
(A/B test) Code discrepancies
Final model
Data discrepancies
Example development process
![Page 17: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/17.jpg)
Avoid dual implementations
Shared Engine
Experimentcode
Productioncode
ProductionExperiment
![Page 18: Paris ML meetup](https://reader033.vdocuments.mx/reader033/viewer/2022051123/586e73cc1a28ab99598b55db/html5/thumbnails/18.jpg)
To be continued...