machine learning logistics

53
© 2017 MapR Technologies 1 Machine Learning Model Management

Upload: ted-dunning

Post on 22-Jan-2018

1.387 views

Category:

Technology


0 download

TRANSCRIPT

Page 1: Machine Learning logistics

© 2017 MapR Technologies 1

Machine Learning Model Management

Page 2: Machine Learning logistics

© 2017 MapR Technologies 2

Contact Information

Ted Dunning, PhD

Chief Application Architect, MapR Technologies

Committer, PMC member, board member, ASF

O’Reilly author

Email [email protected] [email protected]

Twitter @Ted_Dunning

Page 3: Machine Learning logistics

© 2017 MapR Technologies 3

Machine Learning Everywhere

Image courtesy Mtell used with permission.Images © Ellen Friedman.

Page 4: Machine Learning logistics

© 2017 MapR Technologies 4

Traditional View

Page 5: Machine Learning logistics

© 2017 MapR Technologies 5

Traditional View: This isn’t the whole story

Page 6: Machine Learning logistics

© 2017 MapR Technologies 6

90% of the effort in successful machine learning isn’t in the training or model dev…

It’s the logistics

Page 7: Machine Learning logistics

© 2017 MapR Technologies 7

Why?

• Just getting the training data is hard

– Which data? How to make it accessible? Multiple sources!

– New kinds of observations force restarts

– Requires a ton of domain knowledge

• The myth of the unitary model

– You can’t train just one

– You will have dozens of models, likely hundreds or more

– Handoff to new versions is tricky

– You have to get run-time to be sure about which is better

Page 8: Machine Learning logistics

© 2017 MapR Technologies 8

What Machine Learning Tool is Best?

• Most successful groups keep several “favorite” machine

learning tools at hand

– No single tool is best in every situation

• The most important tool is a platform that supports logistics well

– Don’t have to do everything at the application level

– Lots of what matters can be handled at the platform level

• A good design for the logistics can make a big difference

Page 9: Machine Learning logistics

© 2017 MapR Technologies 9

Some Gotchas

• Ops-oriented people will not “get it” regarding modeling

subtleties

• Data scientists will not “get it” regarding operational realities

• Therefore, modelers have to deliver self-contained models

• And, ops has to provide pre-wired structure

Page 10: Machine Learning logistics

© 2017 MapR Technologies 10

Rendezvous Architecture

Input Scores

RendezvousModel 1

Model 2

Model 3

request

response

Results

Page 11: Machine Learning logistics

© 2017 MapR Technologies 11

Rendezvous to the Rescue: Better ML Logistics

• Stream-1st architecture is a powerful approach with surprisingly

widespread advantages

– Innovative technologies emerging to for streaming data

• Microservices approach provides flexibility

– Streaming supports microservices (if done right)

• Containers remove surprises

– Predictable environment for running models

Page 12: Machine Learning logistics

© 2017 MapR Technologies 12

Rendezvous: Mainly for Decisioning Engines

• Decisioning models

– Looking for a “right answer”

– Simpler than reinforcement learning

• Examples include:

– Fraud detection

– Predictive analytics / market prediction

– Churn prediction (as in telecommunications)

– Yield optimization

– Deep learning in form of speech or image recognition, in some cases

Page 13: Machine Learning logistics

© 2017 MapR Technologies 13

Why Stream?

Munich surfing wave Image © 2017 Ellen Friedman

Page 14: Machine Learning logistics

© 2017 MapR Technologies 14

Stream-1st Architecture: Basis for MicroServices

Stream instead of database as the shared “truth”

POS 1..n

Fraud detector

Last card use

Updater

Card analytics

Other

card activity

Image © 2016 Ted Dunning & Ellen Friedman from Chap 6 of O’Reilly book Streaming Architecture used with permission

Page 15: Machine Learning logistics

© 2017 MapR Technologies 15

Streaming Isolates Services

streamData

sourceConsumer

Page 16: Machine Learning logistics

© 2017 MapR Technologies 16

With MapR, Geo-Distributed Data Appears Local

stream

streamData

source

Consumer

Page 17: Machine Learning logistics

© 2017 MapR Technologies 17

With MapR, Geo-distributed Data Appears Local

stream

streamData

source

ConsumerGlobal Data Center

Regional Data Center

Page 18: Machine Learning logistics

© 2017 MapR Technologies 18

Features of Good Streaming

• It is Persistent– Messages stick around for other consumers

– Consumers don’t affect producers

– Consumer doesn’t have to be online when message arrives

• It is Performant– You don’t have to worry if a stream can keep up

• It is Pervasive– It is there whenever you need it, no need to deploy anything

– How much work is it to create a new file? Why harder for a stream?

Page 19: Machine Learning logistics

© 2017 MapR Technologies 19

Stream transport supports microservices

Page 20: Machine Learning logistics

© 2017 MapR Technologies 20

But we talked about decision engines?!?

Page 21: Machine Learning logistics

© 2017 MapR Technologies 21

What We Ultimately Want

request

responseModel

Page 22: Machine Learning logistics

© 2017 MapR Technologies 22

But This Isn’t The Answer

Model 1

request

response

Load

balancerModel 2

Model 3

Page 23: Machine Learning logistics

© 2017 MapR Technologies 23

First Try with Streams

Input

Model 1

Model 2

Model 3

request

response?

Page 24: Machine Learning logistics

© 2017 MapR Technologies 24

First Rendezvous

Input Scores

RendezvousModel 1

Model 2

Model 3

request

response

Results

Page 25: Machine Learning logistics

© 2017 MapR Technologies 25

Some Key Points

• Note that all models see identical inputs

• All models run in production setting

• All models send scores to same stream

• The rendezvous server decides which scores to ignore

• Roll forward, roll back, correlated comparison are all now trivial

Page 26: Machine Learning logistics

© 2017 MapR Technologies 26

Reality Check, Injecting External State

Model 1

Model 2

Model 3

request

Raw

Add external

dataInput

Database

The world

Page 27: Machine Learning logistics

© 2017 MapR Technologies 27

Recording Raw Data (as it really was)

InputScores

Decoy

Model 2

Model 3

Archive

Page 28: Machine Learning logistics

© 2017 MapR Technologies 28

Quality & Reproducibility of Input Data is Important!

• Recording raw-ish data is really a big deal

– Data as seen by a model is worth gold

– Data reconstructed later often has time-machine leaks

– Databases were made for updates, streams are safer

• Raw data is useful for non-ML cases as well (think flexibility)

• Decoy model records training data as seen by models under

development & evaluation

Page 29: Machine Learning logistics

© 2017 MapR Technologies 29

Canary for Comparison

Real

model∆

Result

Canary

Decoy

Archive

Input

Page 30: Machine Learning logistics

© 2017 MapR Technologies 30

What Does the Canary Do?

• The canary is a real model, but is very rarely updated

• The canary results are almost never used for decisioning

• The virtue of the canary is stability

• Comparing to the canary results gives insight into new models

Page 31: Machine Learning logistics

© 2017 MapR Technologies 31

Isolated Development With Stream Replication

Model 1

Model 2

Model 3

request

Raw

Add

external

data

Input

Internal 1

Internal 2

Internal 3

The world

Model 4

Raw

New

external

data

Input

Internal 4

Production

Development

Page 32: Machine Learning logistics

© 2017 MapR Technologies 32

Scores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw

Page 33: Machine Learning logistics

© 2017 MapR Technologies 33

ResultsRendezvousScores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw

Page 34: Machine Learning logistics

© 2017 MapR Technologies 34

MetricsMetrics

ResultsRendezvousScores

ArchiveDecoy

m1

m2

m3

Features / profiles

Input Raw

Page 35: Machine Learning logistics

© 2017 MapR Technologies 35

Models in production live in the real world:

Conditions may (will) change

Page 36: Machine Learning logistics

© 2017 MapR Technologies 36

Not Such Bad Ideas

• Keep models running “in the wings” – Don’t wait until conditions change to start building the next model

– Keep new short-history models ready to roll, some graybeards as well

• Hot hand-off– With rendezvous: just stop ignoring the new best model

• Deploy a canary server– Keep an old model active as a reference

– If it was 90% correct, difference with any better model should be small

– Score distribution should be roughly constant

Page 37: Machine Learning logistics

© 2017 MapR Technologies 37

Correlated Comparison of Score Quantiles

Page 38: Machine Learning logistics

© 2017 MapR Technologies 38

Sample Model Cascade

A

B

Fraud

Fraud

Clean

Clean

Fraud

Assume that finding more frauds is all we care to do

Page 39: Machine Learning logistics

© 2017 MapR Technologies 39

Some Data

Page 40: Machine Learning logistics

© 2017 MapR Technologies 40

Consisting of Type 1

Page 41: Machine Learning logistics

© 2017 MapR Technologies 41

And Type 2

Page 42: Machine Learning logistics

© 2017 MapR Technologies 42

Sample Model Cascade

A

B

Fraud

Fraud

Clean

Clean

Fraud

Good with type 1

Good with type 2

Page 43: Machine Learning logistics

© 2017 MapR Technologies 43

Baseline Conditions

• Model A

– 80% recall on type 1, 0% recall on type 2 (40% net)

• Model B

– 0% recall on type 1, 80% recall on type 2 (40% net)

• Combined

– No overlap in responses

– 80% recall on type 1 (due to model A)

– 80% recall on type 2 (due to model B)

– 80% recall overall

Page 44: Machine Learning logistics

© 2017 MapR Technologies 44

“New and Improved”

• Suppose model A is “improved”

– Before: 80% recall on type 1, 0% recall on type 2 (40% net)

– After: 40% recall on type 1, 100% also on type 2 (70% net)

• Combined after change

– Huge overlap in responses

– 40% recall on type 1 (due to model A)

– 100% recall on type 2 (due to model A)

– Model B has no effect

– 70% recall overall

Page 45: Machine Learning logistics

© 2017 MapR Technologies 45

Coupling Paradox

Page 46: Machine Learning logistics

© 2017 MapR Technologies 46

Is There Any Hope?

• This kind of problem is HARD

– Do your competitor’s and your own marketing model couple?

• Where possible, use ensembles instead of cascades

– Not as simple as it sounds

• Where possible, deploy composite models as units

– Not as simple as it sounds

• Always measure everything!

Page 47: Machine Learning logistics

© 2017 MapR Technologies 47

How to Do Better

• Data + the right question + domain knowledge matter!

• Prioritize – put serious effort into infrastructure

– DataOps requires more than just data science

• Persist – use streams to keep data around

• Measure – everything, and record it

• Meta-analyze – understand and see what is happening

• Containerize – make deployment repeatable, easy

• Oh… don’t forget to do some machine learning, too

Page 48: Machine Learning logistics

© 2017 MapR Technologies 48

Additional Resources

O’Reilly report by Ted Dunning & Ellen Friedman © March 2017

Read free courtesy of MapR:

https://mapr.com/geo-distribution-big-data-and-analytics/

O’Reilly book by Ted Dunning & Ellen Friedman

© March 2016

Read free courtesy of MapR:

https://mapr.com/streaming-architecture-using-

apache-kafka-mapr-streams/

Page 49: Machine Learning logistics

© 2017 MapR Technologies 49

Additional Resources

O’Reilly book by Ted Dunning & Ellen Friedman

© June 2014

Read free courtesy of MapR:

https://mapr.com/practical-machine-learning-

new-look-anomaly-detection/

O’Reilly book by Ellen Friedman & Ted Dunning

© February 2014

Read free courtesy of MapR:

https://mapr.com/practical-machine-learning/

Page 50: Machine Learning logistics

© 2017 MapR Technologies 50

Additional Resources

by Ellen Friedman 8 Aug 2017 on MapR blog:

https://mapr.com/blog/tensorflow-mxnet-caffe-h2o-which-ml-best/

by Ted Dunning 13 Sept 2017 in

InfoWorld:

https://www.infoworld.com/article/3223

688/machine-learning/machine-

learning-skills-for-software-

engineers.html

Page 51: Machine Learning logistics

© 2017 MapR Technologies 51

New book: Machine Learning Logistics

Model Management in the Real World

O’Reilly book by Ellen Friedman & Ted Dunning © Sept 2017

Pre-register for a free pdf copy of book when it becomes available 26th

September, courtesy of MapR

http://info.mapr.com/2017_Content_Machine-Learning-

Logistics_eBook_Prereg_RegistrationPage.html

Going to Strata Data NYC? Book will be released 26 Sept 2017:

Visit MapR booth for free book signings or to talk about logistics

Page 52: Machine Learning logistics

© 2017 MapR Technologies 52

Please support women in tech – help build

girls’ dreams of what they can accomplish

© Ellen Friedman 2015#womenintech #datawomen

Page 53: Machine Learning logistics

© 2017 MapR Technologies 53

Q&A

@mapr

[email protected]

ENGAGE WITH US

@ Ted_Dunning