hadoop in three use cases

2 December 2011

Hadoop in Three Use CasesJoey Echeverria | Solutions [email protected] | @fwiffo

©2011 Cloudera, Inc. All Rights Reserved.2

About Joey

• Solutions Architect• 6 months• 3+ years• Local

Cloudera’s Distribution including Apache Hadoop

Copyright 2011 Cloudera Inc. All rights reserved3

Coordination

Data IntegrationFast Read/Write

Access

Languages / Compilers

Workflow Scheduling Metadata

APACHE ZOOKEEPER

APACHE FLUME*, APACHE SQOOP* APACHE HBASE

APACHE PIG, APACHE HIVE

APACHE OOZIE* APACHE OOZIE* APACHE HIVE

File System Mount UI Framework SDKFUSE-DFS HUE HUE SDK

*currently under incubation in the Apache Software Foundation

Extract, Transform, and Load



ETL before Hadoop

Difficult to maintain, not scalable

Logs

Files

Relational Databases

Enterprise Data Warehouse

Custom ETL Scripts


ETL before Hadoop

May be scalable, expensive

Logs

Files


Enterprise Data Warehouse SQL:

raw table → warehouse tables


ETL with Hadoop

Managed, flexible, scalable

Logs

Files


Enterprise Data Warehouse

Steps


2. Process

1. In

3. Out

Flume



ETL with Hadoop


Logs

Files


Enterprise Data WarehouseFlume

HDFS


HDFS


Client

NameNode

DataNode 01

DataNode 05

DataNode 09

DataNode 02

DataNode 06

DataNode 10

DataNode 03

DataNode 07

DataNode 11

DataNode 04

DataNode 08

DataNode 12

open(“file.txt”)

02, 06, 10

data

data data

HDFS

• Distributed• Replication• Bulk I/O• Fault tolerant• Scalable• Append only• Not POSIX



ETL with Hadoop


Logs

Files


Enterprise Data WarehouseFlume HDFS

FUSE-DFS


FUSE-DFS

• FUSE– User space– File systems

• FUSE-DFS– /hdfs– Mostly transparent



ETL with Hadoop


Logs

Files



FUSE-DFS

HDFS

Sqoop


Sqoop


• SQL to Hadoop• Parallel import• File formats


ETL with Hadoop


Logs

Files



FUSE-DFS

Sqoop

HDFS

Pig


Pig

• Scripting language• Generates MapReduce jobs• Perl for Hadoop• Great for ETL


A = LOAD 'data' USING PigStorage() AS (f1:int, f2:int, f3:int);B = GROUP A BY f1;C = FOREACH B GENERATE COUNT ($0);DUMP C;


ETL with Hadoop


Logs

Files



FUSE-DFS

Sqoop

HDFS

Pig

Sqoop with connectors


Sqoop with connectors

• MySQL*• PostgreSQL*• Teradata*• Netezza*• Oracle*• Couchbase*• Microsoft SQL Server• VoltDB


*Cloudera certified connector


ETL with Hadoop


Logs

Files



FUSE-DFS

Sqoop

HDFS

Sqoop

Pig

Recommendations



Recommendations with Hadoop

Logs


Web Application

CUSTOMERS

Flume




Logs


Flume

Web Application

CUSTOMERS

HDFS




Logs


Flume HDFS

Web Application

CUSTOMERS

Sqoop




Logs


Flume

Sqoop

HDFS

Web Application

CUSTOMERS

Pig




Logs


Flume

Sqoop

HDFS

Pig

Web Application

CUSTOMERS

Mahout


Mahout

• Scalable machine learning algorithms– Collaborative Filtering– User and Item based recommenders– K-Means, Fuzzy K-Means clustering– Mean Shift clustering– Singular value decomposition– Complementary Naive Bayes classifier …




Logs


Flume

Sqoop

HDFS

Pig

Web Application

CUSTOMERS

Mahout

MapReduce


MapReduce


toOne()

toOne()

toOne()

:1

:1

:1

:1

:1

:1

:1

:1

:1

count():[1,1,1,1]

:[1,1]

:[1,1]

:[1]

count()

:4

:2

:2

:1

shufflemap reduce

MapReduce

• Distributed• Code to data• Reliable• Scalable




Logs


Flume

Sqoop

HDFS

Pig

Web Application

CUSTOMERS

Mahout MapReduce Pig

Oozie


Oozie

• Workflows• Coordinator

– Triggers




Logs


Flume

Sqoop

HDFS

Pig

Web Application

CUSTOMERS


Oozie

HBase


HBase

• Key/value store

• Data stored in HDFS

• Access model is get/put/del– Plus range scans and versions

• Random reads and writes for Hadoop




Logs


Flume

Sqoop

HDFS

Pig

Web Application

CUSTOMERS


OozieHBase

Business Intelligence



Business Intelligence with Hadoop

Logs


BI / Analytics

ANALYSTS

Flume




Logs


Flume

BI / Analytics

ANALYSTS

HDFS




Logs


Flume HDFS

BI / Analytics

ANALYSTS

Sqoop




Logs


Flume

Sqoop

HDFS

BI / Analytics

ANALYSTS

Hive


Hive

• Data warehouse• Ad-hoc queries

– Not real-time (minutes)

• SQL• Tables• Joins




Logs


Flume

Sqoop

HDFS

BI / Analytics

ANALYSTS

Hive

MapReduce




Logs


Flume

Sqoop

HDFS

BI / Analytics

ANALYSTS

Hive MapReduce

Oozie




Logs


Flume

Sqoop

HDFS

BI / Analytics

ANALYSTS

Hive

Oozie

MapReduce

HBase




Logs


Flume

Sqoop

HDFS HBase

BI / Analytics

ANALYSTS

Hive

Oozie

MapReduce

Hive


Hive for Business Intelligence

• JDBC– JasperReports*– Pentaho*

• ODBC– MicroStrategy*^


* Vender certified connector^ Cloudera certified connector



Logs


Flume

Sqoop

HDFS Hive HBase

BI / Analytics

ANALYSTS

Hive

Oozie

MapReduce

CDH


Coordination

Data IntegrationFast Read/Write

Access

Languages / Compilers

Workflow Scheduling Metadata

APACHE ZOOKEEPER

APACHE FLUME*, APACHE SQOOP* APACHE HBASE

APACHE PIG, APACHE HIVE

APACHE OOZIE* APACHE OOZIE* APACHE HIVE

File System Mount UI Framework SDKFUSE-DFS HUE HUE SDK

*currently under incubation in the Apache Software Foundation

What’s next?

• Cloudera Training Videos• CDH Virtual Machines• Hadoop: The Definitive Guide, 2nd Edition• Cloudera University

– Developer Training in Columbia, MD• Dec 13-16, Feb 13-16

– Administrator Training in Herndon, VA• Jan 4-6

– Private Training


http://www.cloudera.com/resources/?type=Training

https://ccp.cloudera.com/display/SUPPORT/Cloudera's+Hadoop+Demo+VM

http://shop.oreilly.com/product/0636920010388.do

http://www.cloudera.com/hadoop-training/

We’re Hiring!

• http://www.cloudera.com/company/careers /• Customer Operations

– Customer Operations Engineer– Customer Operations Tools Developer

• Customer Solutions– Solutions Architect

• Engineering– Senior Data Integration Developer– Senior Distributed Systems Engineer– Senior UI Engineer– Software Quality Engineer– Technical Writer

• IT/Operations– Systems Administrator


http://www.cloudera.com/company/careers/

http://www.cloudera.com/company/careers/

hadoop in three use cases

Documents

apache sqoop

load4 copyright

connectors25 copyright

out8 copyright

flume10 copyright

sqoop19 copyright

pig22 copyright

recommendations28 copyright