mt30 best practices for data lake adoption

MT30

Best practices: Data lake adoption

Matt Maccaux, Global Big Data Practice Lead

2Dell - Internal Use - Confidential

Agenda

• Two models for big data

• Big data anti-patterns

• Big data best practice

• How to get started?

• Your questions


Two models for big data

Exploratory analytics

• Full data set – batch

• Explore, test, refine,

iterate

• The output is an algorithm

that will be integrated into

new or existing

applications.

Operationalization

• Limited data set –

Streaming

• The algorithm is integrated

into applications that drive

business decisions.

Big data anti-patterns

Dell - Internal Use - Confidential

© Copyright 2016 EMC Corporation. All rights reserved.

Big data best practices



me:~>_

CONTINUUM



Hadoop

Spark

Tableau

Python

TOOL CATALOG

Customer

Alert

Bills

Social

DATACATALOG

Duration

Performance

Normal

Analytics Request Portal

NONSampleData

SampleData



Data Lake

Discover/Map

Transform

Organize/Tag

CATALOG AND PROVISION

ENTERPRISE LOG ANALYSIS

Virtualisation

http://www.google.com/url?url=http://bigdatavendors.com/top.php&rct=j&frm=1&q=&esrc=s&sa=U&ei=G3U1VdaHDMueyATTk4GoDQ&ved=0CBgQ9QEwAQ&usg=AFQjCNGPF3lS-Cr1fgjnUGhiCaTZrf6Mqg

http://www.google.com/url?url=http://bigdatavendors.com/top.php&rct=j&frm=1&q=&esrc=s&sa=U&ei=G3U1VdaHDMueyATTk4GoDQ&ved=0CBgQ9QEwAQ&usg=AFQjCNGPF3lS-Cr1fgjnUGhiCaTZrf6Mqg

http://www.google.com/url?url=http://hortonworks.com/blog/the-future-of-apache-storm/&rct=j&frm=1&q=&esrc=s&sa=U&ei=gHY1VcLYC4mPyASYjoHQAQ&ved=0CBwQ9QEwAw&usg=AFQjCNHpG5A9AuTmKKwvOqilSl1ocMQmFQ

http://www.google.com/url?url=http://hortonworks.com/blog/the-future-of-apache-storm/&rct=j&frm=1&q=&esrc=s&sa=U&ei=gHY1VcLYC4mPyASYjoHQAQ&ved=0CBwQ9QEwAw&usg=AFQjCNHpG5A9AuTmKKwvOqilSl1ocMQmFQ

http://www.google.com/url?url=http://www.hadoopsphere.com/2015/03/hadoop-high-5-with-ibms-anjul-bhambhri.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=mnY1VamjCI2uyATWooCIAw&ved=0CB4Q9QEwBA&usg=AFQjCNEwNc97BzzIMTHPvCy419nHjKOLhg

http://www.google.com/url?url=http://www.hadoopsphere.com/2015/03/hadoop-high-5-with-ibms-anjul-bhambhri.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=mnY1VamjCI2uyATWooCIAw&ved=0CB4Q9QEwBA&usg=AFQjCNEwNc97BzzIMTHPvCy419nHjKOLhg

http://www.google.com/url?url=http://spark-summit.org/east&rct=j&frm=1&q=&esrc=s&sa=U&ei=0HY1VfipH4n2yQTt6YC4Aw&ved=0CBgQ9QEwAA&usg=AFQjCNG_COnNEYyI9VRZC4SyqIc9l96Mrg

http://www.google.com/url?url=http://spark-summit.org/east&rct=j&frm=1&q=&esrc=s&sa=U&ei=0HY1VfipH4n2yQTt6YC4Aw&ved=0CBgQ9QEwAA&usg=AFQjCNG_COnNEYyI9VRZC4SyqIc9l96Mrg

http://www.google.com/url?url=http://hortonworks.com/hadoop/kafka/&rct=j&frm=1&q=&esrc=s&sa=U&ei=JRJRVYm7GYqyyATp3YD4Bw&ved=0CBYQ9QEwAA&usg=AFQjCNFpxjAEtSPHO7THDuSpXL3-tJBnXw

http://www.google.com/url?url=http://hortonworks.com/hadoop/kafka/&rct=j&frm=1&q=&esrc=s&sa=U&ei=JRJRVYm7GYqyyATp3YD4Bw&ved=0CBYQ9QEwAA&usg=AFQjCNFpxjAEtSPHO7THDuSpXL3-tJBnXw

http://www.bizjournals.com/sanjose/blog/2012/06/san-joses-skytree-2-others-win-at.html?s=image_gallery

http://www.bizjournals.com/sanjose/blog/2012/06/san-joses-skytree-2-others-win-at.html?s=image_gallery

http://www.google.com/url?url=http://www.sitepoint.com/efficient-chinese-search-elasticsearch/&rct=j&frm=1&q=&esrc=s&sa=U&ei=UXU1Vf7EBI-gyQSz94CQAQ&ved=0CBoQ9QEwAQ&usg=AFQjCNFR3dtP-tI7IxXrTSGFmJdeJNe0Zg

http://www.google.com/url?url=http://www.sitepoint.com/efficient-chinese-search-elasticsearch/&rct=j&frm=1&q=&esrc=s&sa=U&ei=UXU1Vf7EBI-gyQSz94CQAQ&ved=0CBoQ9QEwAQ&usg=AFQjCNFR3dtP-tI7IxXrTSGFmJdeJNe0Zg

http://www.google.com/url?url=http://pixshark.com/kibana.htm&rct=j&frm=1&q=&esrc=s&sa=U&ei=i3U1Va7LNMWXyATCroDoAg&ved=0CBwQ9QEwAw&usg=AFQjCNG3dAT7sE9gEILc-jRnfJLwJUeXWQ

http://www.google.com/url?url=http://pixshark.com/kibana.htm&rct=j&frm=1&q=&esrc=s&sa=U&ei=i3U1Va7LNMWXyATCroDoAg&ved=0CBwQ9QEwAw&usg=AFQjCNG3dAT7sE9gEILc-jRnfJLwJUeXWQ

http://www.google.com/url?url=http://altaplana.com/clients.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=-XY1VdXhJoKzyQTW_IGwCg&ved=0CBYQ9QEwAA&usg=AFQjCNF9_URNTrV0E64PWy3S79Ym5Rtw7A

http://www.google.com/url?url=http://altaplana.com/clients.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=-XY1VdXhJoKzyQTW_IGwCg&ved=0CBYQ9QEwAA&usg=AFQjCNF9_URNTrV0E64PWy3S79Ym5Rtw7A

http://www.tpimagazine.com/siteimage/scale/800/600/353400.png

http://www.tpimagazine.com/siteimage/scale/800/600/353400.png

http://www.google.com/url?url=http://en.wikipedia.org/wiki/Greenplum&rct=j&frm=1&q=&esrc=s&sa=U&ei=MOs4VevPJJbfsASAmIHoBg&ved=0CDwQ9QEwEw&usg=AFQjCNEO7zx6us020SM17YnIuR5LKppk2Q

http://www.google.com/url?url=http://en.wikipedia.org/wiki/Greenplum&rct=j&frm=1&q=&esrc=s&sa=U&ei=MOs4VevPJJbfsASAmIHoBg&ved=0CDwQ9QEwEw&usg=AFQjCNEO7zx6us020SM17YnIuR5LKppk2Q

http://www.google.com/url?url=http://www.oracle.com/technetwork/database/exadata/overview/index.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=hus4VeynIriKsQSF6oCAAg&ved=0CBYQ9QEwAA&usg=AFQjCNGNRS-bjm4SUXibEDLFk9H5RhmgHg

http://www.google.com/url?url=http://www.oracle.com/technetwork/database/exadata/overview/index.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=hus4VeynIriKsQSF6oCAAg&ved=0CBYQ9QEwAA&usg=AFQjCNGNRS-bjm4SUXibEDLFk9H5RhmgHg

http://www.google.com/url?url=http://logo-kid.com/pivotal-hd-logo.htm&rct=j&frm=1&q=&esrc=s&sa=U&ei=n-w4VcvYB4y1sATO7YGoBg&ved=0CBYQ9QEwAA&usg=AFQjCNEeQD_s76YWxnq931MgAcQ-F4RZAg

http://www.google.com/url?url=http://logo-kid.com/pivotal-hd-logo.htm&rct=j&frm=1&q=&esrc=s&sa=U&ei=n-w4VcvYB4y1sATO7YGoBg&ved=0CBYQ9QEwAA&usg=AFQjCNEeQD_s76YWxnq931MgAcQ-F4RZAg

http://www.google.com/url?url=http://www.cloudera.com/content/cloudera/en/campaign/introducing-impala.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=_e04VeSTLc61sATfrYHYCw&ved=0CBYQ9QEwAA&usg=AFQjCNF28VkVUrV07WSP2qs8S0vERAlC5w

http://www.google.com/url?url=http://www.cloudera.com/content/cloudera/en/campaign/introducing-impala.html&rct=j&frm=1&q=&esrc=s&sa=U&ei=_e04VeSTLc61sATfrYHYCw&ved=0CBYQ9QEwAA&usg=AFQjCNF28VkVUrV07WSP2qs8S0vERAlC5w

http://www.google.com/url?url=http://hortonworks.com/hadoop/hive/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Oe44VbqmF-u1sATpnIBA&ved=0CBYQ9QEwAA&usg=AFQjCNGhlkNn-JB-ODmCFyT-65304tlSiA

http://www.google.com/url?url=http://hortonworks.com/hadoop/hive/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Oe44VbqmF-u1sATpnIBA&ved=0CBYQ9QEwAA&usg=AFQjCNGhlkNn-JB-ODmCFyT-65304tlSiA

https://www.google.com/url?url=https://hadoopi.wordpress.com/2014/10/25/use-spark-sql-on-sql-developer/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Ze44VeatNteHsQS3voDoBQ&ved=0CBYQ9QEwAA&usg=AFQjCNEZIaFwJhhkthDwP4wcHl-y_k5hQg

https://www.google.com/url?url=https://hadoopi.wordpress.com/2014/10/25/use-spark-sql-on-sql-developer/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Ze44VeatNteHsQS3voDoBQ&ved=0CBYQ9QEwAA&usg=AFQjCNEZIaFwJhhkthDwP4wcHl-y_k5hQg

http://www.google.com/url?url=http://prestodb.io/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Du84VYWhJuS1sAS8k4DgBQ&ved=0CBYQ9QEwAA&usg=AFQjCNHXjNHV9FjsbsPTTOXu6FbrKW-CTg

http://www.google.com/url?url=http://prestodb.io/&rct=j&frm=1&q=&esrc=s&sa=U&ei=Du84VYWhJuS1sAS8k4DgBQ&ved=0CBYQ9QEwAA&usg=AFQjCNHXjNHV9FjsbsPTTOXu6FbrKW-CTg

https://www.google.com/url?url=https://gigaom.com/2014/09/16/sql-on-hadoop-tech-apache-drill-is-ready-to-use-and-part-of-maprs-distro/&rct=j&frm=1&q=&esrc=s&sa=U&ei=c_E4Vaj7G6K1sATfh4HAAw&ved=0CBYQ9QEwAA&usg=AFQjCNHepqA2fhbLGHvLuT9n6-Yfe8PCiw

https://www.google.com/url?url=https://gigaom.com/2014/09/16/sql-on-hadoop-tech-apache-drill-is-ready-to-use-and-part-of-maprs-distro/&rct=j&frm=1&q=&esrc=s&sa=U&ei=c_E4Vaj7G6K1sATfh4HAAw&ved=0CBYQ9QEwAA&usg=AFQjCNHepqA2fhbLGHvLuT9n6-Yfe8PCiw



Virtualised Compute Pool



Data Pool

Meta

-dat

a T

aggi

ng

G

o

v

e

r

n

a

n

c

e

A

n

o

n

y

m

i

s

e

E

n

c

r

y

p

t

i

o

n

Pooln

Pooln

Pooln

Copy



CD \>_

CONTINUUM

Data Pool

G

o

v

e

r

n

a

n

c

e

A

n

o

n

y

m

i

s

e

E

n

c

r

y

p

t

i

o

n

Pooln

Pooln

Pooln

Copy



How to get started?

Big Data Technology Advisory

• Interview stakeholders including business users and technical/functional

experts

• Document requirements and gaps

• Define a future-state reference architecture

• Provide a plan/roadmap for implementation

mt30 best practices for data lake adoption

Documents