hadoop world 2011: integrating hadoop with enterprise rdbms using apache sqoop and other tools - guy...

November 2011

Apache Sqoop (Incubating)Integrating Hadoop with Enterprise RDBS – Part I

Arvind Prabhakar (arvind at apache dot org)

Apache Sqoop Committer and Software Engineer at Cloudera

1

©2011 Quest Software, Inc. All rights reserved..

Hadoop Data Processing

1

2



2

3



3

4



4

5


In This Session…

How Sqoop Works

Roadmap

5

6


Data Import

6

7


Data Import

7

8


Data Import

8

9


Data Import

9

10


Data Import

10

11


Sqoop Overview

11

12


Pre-processing

12

13


Code Generation

13

14


Type Mapping

14

15


Data Transfer

15

16


Data Transfer

16

17


Data Transfer

17

18


Post-Processing

18

19


Sqoop Connectors

Oracle – Developed by Quest Software

Couchbase – Developed by Couchbase

Netezza – Developed by Cloudera

Teradata – Developed by Cloudera

SQL Server – Developed by Microsoft

Microsoft PDW – Developed by Microsoft

Volt DB – Developed by Volt DB

19

20


Sqoop Roadmap

SQOOP-365: Proposal for Sqoop 2.0

• https://issues.apache.org/jira/browse/SQOOP-365

Highlights

• Sqoop as a Service

• Connections as First Class Objects

• Role based Security

20

https://issues.apache.org/jira/browse/SQOOP-365



21


Sqoop 2 Architecture (proposed)

21

22


For More Information

Website:

http://incubator.apache.org/sqoop/

Mailing Lists:

[email protected]

[email protected]

•Issue Tracker:

http://issues.apache.org/jira/browse/SQOOP

22



mailto:[email protected]














http://issues.apache.org/jira/browse/SQOOP

23


Thank You!

Q & A will be after part II of this session.

23


Guy Harrison, Quest Software

Integrating Hadoop with Enterprise

RDBMS Using Apache SQOOP and

Other Tools

25


Introductions

26


28


Agenda

• Scenarios for RDBMS-Hadoop interaction

• Case study: Quest extension to SQOOP

• Other RDBMS-Hadoop integrations

29


Hadoop meets RDBMS – scenarios

Scenario #1: Reference data in RDBMS

CUSTOMERS

WEBlOGS

PRODUCTS

HDFS

RDBMS

Scenario #2: Hadoop for off-line analytics

CUSTOMERS

PRODUCTS

HDFS

RDBMS

SALES

HISTORY

Scenario #3: MapReduce output to RDBMS

WEBLOGS

SUMMARY

RDBMS

DB QUERY

TOOL

WEBlOGS

HDFS

Scenario #4: Hadoop as RDBMS “active archive”

SALES 2011

HDFS

RDBMS

QUERY

TOOL

SALES 2010

SALES 2009

SALES 2008

SALES 2009

SALES 2008

34


Case Study: extending SQOOP for Oracle

35


SQOOP extensibility

• SQOOP implements a generic approach to

RDBMS/Hadoop data transfer

• But database optimization is highly platform specific

• Each RDBMS has distinct optimizations strategies

• For Oracle, optimization requires:

• Bypassing Oracle caching layers

• Avoiding Oracle optimizer meddling

• Exploiting Oracle metadata to balance mapper load

CACHE

ORACLE TABLE

Reading from Oracle – default SQOOP

Index block Index block

RANGE SCAN

MAPPER

ORACLE SESSSION

ID > 0 and ID < MAX/2

MAPPER

ORACLE SESSION

ID > MAX/2


RANGE SCAN


Oracle

SALES

table

HDFS

Hadoop Mapper

Hadoop Mapper

Hadoop Mapper

Hadoop Mapper

Oracle – parallelism gone bad (1)

ORACLE

TABLEHDFS

HADOOP

MAPPER

HADOOP

MAPPER

HADOOP

MAPPER

HADOOP

MAPPER

Oracle – parallelism gone bad (2)

ORACLE

TABLEHDFS

Ideal architecture

HADOOP

MAPPERORACLE

SESSION

HADOOP

MAPPERORACLE

SESSION

HADOOP

MAPPERORACLE

SESSION

HADOOP

MAPPERORACLE

SESSION

40


Design goals

• Partition data based on physical storage

• By-pass Oracle buffering

• By-pass Oracle parallelism

• Do not require or use indexes

• Never read the same data block more than once

• Support Oracle datatypes

41


Import Throughput

0

1,000

2,000

3,000

4,000

5,000

6,000

7,000

0 5 10 15 20 25 30 35

Ela

psed

Tim

e (

ms)

Number of mappers

SQOOP

SQOOP with Quest Connector

42


80.84

89.72

98.95

99.08

98.71

0 20 40 60 80 100

Elasped time

CPU Time

Network round trips

IO requests

IO time

Pct reduction

16 mappers, 50M rows, 50 GB clustered data

43


Export Throughput

500

1,000

1,500

2,000

2,500

3,000

0 5 10 15 20 25

Seco

nd

s

No of mappers

SQOOP

SQOOP with Quest Connect

44


Export load

0

5000

10000

15000

20000

25000

30000

0 5 10 15 20 25 30

Data

base t

ime (

s)

No of mappers

SQOOP

SQOOP with Quest connect

45


Working with the SQOOP framework

• SQOOP lets you concentrate on the RDBMS logic, not

the Hadoop plumbing:

• Extend ManagerFactory (what to handle)

• Extend ConnManager (DB connection and metadata)

• For imports:

• Extend DataDrivenDBInputFormat (gets the data)

• Data allocation (getSplits())

• Split serialization (“io.serializations” property)

• Data access logic (createDBRecordReader(), getSelectQuery())

• Implement progress (nextKeyValue(), getProgress())

• Similar procedure for extending exports

46


Extensions to native SQOOP

• MERGE

functionality

• Update if

exists, insert

otherwise

• Hive

connector

• Source defined as

HQL query rather

than HDFS

directory

• Eclipse UI

47


Availability

• Apache licensed source available from :

https://github.com/QuestSoftwareTCD/OracleSQOOPconnector

• Download from (Quest):

http://www.quest.com/hadoop/

• Download from (Cloudera):

http://ccp.cloudera.com/display/SUPPORT/Downloads

https://github.com/QuestSoftwareTCD/OracleSQOOPconnector





48


Other SQOOP connectors

• Microsoft SQL Server:

• http://www.microsoft.com/download/en/details.aspx?id=27584

• Teradata:

• https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Teradata+User+Guide

%2C+version+1.0-beta-u4

• Microstrategy:

• https://ccp.cloudera.com/display/con/MicroStrategy+Free+Download+License+Agreem

ent

• Nettezza:

• https://ccp.cloudera.com/display/con/Netezza+Free+Download+License+Agreement

• VoltDB:

• http://voltdb.com/company/blog/sqoop-voltdb-export-and-hadoop-integration

http://www.microsoft.com/download/en/details.aspx?id=27584

http://www.microsoft.com/download/en/details.aspx?id=27584

https://ccp.cloudera.com/display/con/Cloudera+Connector+for+Teradata+User+Guide,+version+1.0-beta-u4







https://ccp.cloudera.com/display/con/MicroStrategy+Free+Download+License+Agreement



https://ccp.cloudera.com/display/con/Netezza+Free+Download+License+Agreement

https://ccp.cloudera.com/display/con/Netezza+Free+Download+License+Agreement

http://voltdb.com/company/blog/sqoop-voltdb-export-and-hadoop-integration











49


Other Hadoop – RDBMS integrations

50


Oracle Big Data Appliance

• 18 Sun X4270 M2 servers

• 48GB per node (864GB total)

• 2x6 Core CPU per node (216 total)

• 12x2TB HDD per node (216 spindles, 864 TB)

• 40Gb/s Infiniband between nodes

• 10Gb/s Ethernet to datacenter

• Apache Hadoop

• Oracle NoSQL

• Oracle loader for Hadoop

• Multi-stage C-optimized unidirectional loader

www.oracle.com/us/bigdata/index.html

http://www.oracle.com/us/bigdata/index.html





ORACLE

EXADATA

ORACLE

EXALOGIC

ORACLE

BIG DATA

APPLIANCE

ORACLE

NOSQL

ORACLE

LOADER

FOR

HADOOPAPACHE

HADOOPORACLE

RDBMS

ORACLE

WEBLOGIC

ORACLE

EXALYTICS

ORACLE

ESSBASE

ORACLE

TIMES TEN

52


Microsoft

53


Hadapt

• Formally HadoopDB – Hadoop/Postgres hybrid

• Postgres servers on data nodes allow for accelerated

(indexed) HIVE queries

• Extensions to the Hive optimizer

http://www.hadapt.com/

http://www.hadapt.com/

54


Greenplum

• SQL based access to HDFS data via in-DB MapReduce

http://www.greenplum.com/sites/default/files/EMC_Greenplum_Hadoop_DB_TB_0.pdf

http://www.greenplum.com/sites/default/files/EMC_Greenplum_Hadoop_DB_TB_0.pdf

55


Toad for Cloud Databases

• Federated SQL queries across

Hive, Hbase, NoSQL, RDBMS

56


Conclusions

• RDBMS-Hadoop interoperability is key to Enterprise

Hadoop adoption

• SQOOP provides a good general purpose framework

for transferring data between any JDBC database and

Hadoop

• We’d like to see it become a standard

• Each RDBMS offers distinct tuning opportunities, so

optimized SQOOP extensions offer real value

• Hadoop-RDBMS integration projects are proliferating

rapidly

57


hadoop world 2011: integrating hadoop with enterprise rdbms using apache sqoop and other tools - guy...

Technology