real-world machine learning - leverage the features of mapr converged data platform

© 2016 MapR Technologies 1© 2016 MapR Technologies 1MapR Confidential © 2016 MapR Technologies

Real-World Machine Learning - Leverage the Features of MapR Converged Data Platform

Mathieu Dumoulin ([email protected])Mateusz Dymczyk ([email protected])Hadoop Summit Tokyo 2016

mailto:[email protected]


© 2016 MapR Technologies 2© 2016 MapR Technologies 2MapR Confidential

Mathieu Dumoulin, Data Engineer

• Master’s degree in text classification on Hadoop at Fujitsu Canada’s Innovation Lab

• In Tokyo, I’ve worked as Data Scientist, Search Engineer and Data Engineer

• My favorite ML libs are Scikit-Learn and H2O

• 日本料理が大好き。とくに鍋としゃぶ

しゃぶです。


Mateusz Dymczyk, Software Engineer

• About me


A Machine Learning Pipeline

Image from scikit-learn.org


… Meets the Real World

Must be integrated with a production system


… Meets the Real WorldData comes from many sourcesmaybe very large

Data isn’t always labeled!



… Meets the Real WorldData comes from many sources,maybe very large

Needs ETL and cleaning

Finding the best algorithm and parameters can use a lot of CPU



… Meets the Real WorldData comes from many sources,maybe very large

Needs ETL and cleaning

Finding the best algorithm and parameters can use a lot of CPU


From production systems? Is it real time?


The predictions are used by another system...


Doing Machine Learning here...


Is very different than here


We don’t have better algorithms, we just have more data.

Peter Norvig, CTO at Google


Machine Learning at scale mattersGrowing number of ML use cases at successful companies

Anomaly Detection

Customer 360Fraud DetectionLog Security

Analysis

Recommender Sensor Data (IoT)

Personalized Offers

Ad Tech


ML at scale matters… but it’s HARD

Ref: http://advancedspark.com/ , https://github.com/fluxcapacitor/pipeline

http://advancedspark.com/

https://github.com/fluxcapacitor/pipeline


There must be a better way...


A platform for big data ML

• ML projects can start simple and show value

• Just work. Integrate with existing systems, and tools

• Integrate common technology, not just YARN

• Easy, unified administration

• Share the cluster (multi-tenancy)

• Keeps your data safe and secure

What’s an ideal big data platform for ML?


MapR Converged Data Platform


MapR Converged Data Platform

Open Source Engines & Tools Commercial Engines & Applications

Utility-Grade Platform Services

Dat

aP

roce

ssin

g

Enterprise StorageMapR-FS MapR-DB MapR Streams

Database Event Streaming

Global Namespace High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy

Search & Others

Cloud & Managed Services

Custom Apps

Unified M

anagement and M

onitoring


Unique MapR features useful for ML

● MapR-FS and NFS mount

● Topologies

● Mirrors and Snapshots

●

●

● Reliability

● Multi-tenancy

● Data Governance

● Security


MapR MCS

● Unified view

● Easy use of features

● REST API and

maprcli utility


NFS MountMount the cluster as a regular folder

$> sudo mount -o hard,nolock ip-10-0-0-110:/mapr /mapr$> ll /mapr/hadoopsummit/total 3drwxr-xr-x. 3 mapr mapr 1 Oct 13 11:21 appsdrwxr-xr-x. 2 mapr mapr 0 Oct 13 11:12 hbasedrwxr-xr-x. 3 root root 1 Oct 13 11:21 installerdrwxr-xr-x. 2 mapr mapr 0 Oct 13 11:14 optdrwxrwxrwx. 2 mapr mapr 1 Oct 14 10:41 tmpdrwxr-xr-x. 6 mapr mapr 4 Oct 14 10:52 userdrwxr-xr-x. 3 mapr mapr 1 Oct 13 11:13 var

© 2014 MapR Technologies 21

MapR NFS and Volumes

[mapr@ip-10-0-0-110 mapr]$ pwd/mapr/hadoopsummit/user/mapr


Match a Volume to a Topology

Match data to nodes or groups of nodes precisely


CRISP-DM Model

● Industry Standard Model

● Full project view, from business idea to production deployment

● Realistic: lots of cycles


MapR Features for Data Understanding

Data Collection:• NFS Mount• POSIX Client• MapR Streams (Kafka API)• MapR DB (HBase API)Data Exploration

• <Insert your favorite tool>


MapR Features for Data Preparation

Data Cleaning and Feature Engineering (ETL):

• NFS Mount, POSIX Client• Snapshots• Streamsets Data Collector w/

MapR support• Apache Spark• <Your favorite tool>


MapR Features for Modeling

MapR does not “do” machine learning, that’s your job!• MapR Filesystem

• NFS mount/POSIX client

• Mirrors and Snapshots

• Topologies

• Use your existing tools


MapR Features for Evaluation

- Collect data- Explore data

• MapR-FS

• Mirrors

• Snapshots

• Support any tools


MapR Features for Deployment

- Collect data- Explore data

• NFS/POSIX client

• Mirrors

• Snapshots

• Microservice model *

• MapR-DB, MapR Streams

• Security* Check out the converged application blueprint : https://www.mapr.com/appblueprint/overview

https://www.mapr.com/appblueprint/overview

https://www.mapr.com/appblueprint/overview


Converged Data Platform Machine Learning• Features that work together to support all phases of real

production ML

• Supports all the tools you know and the state of the art

frameworks

• Easier to manage, more robust and secure.

• MapR is made for the enterprise


Demo: ML with H2O on MapR


Demo of H2O on MapR: Features in Action


Q & A@mapr

[email protected]

Engage with us!

mapr-technologies




real-world machine learning - leverage the features of mapr converged data platform

Technology