understanding apache hadoop - neudesic...• employs contributors to project apache hadoop •...

© Copyright 2014, Neudesic. All rights reserved.

Understanding Apache Hadoop

Presented By: Orion GebremedhinBI/Big Data Solution Partner , Neudesic LLC.

Data Platform VTSP, Microsoft Corp.

@OrionGM


Topics Covered

• Fundamentals of Hadoop

• Major Hadoop Distributions


3

Big Data = Hadoop?

* A Modern Data Architecture with Apache Hadoop, Hortonworks Inc. 2014


4

The Fundamentals of Hadoop

• Hadoop evolved directly from

commodity scientific supercomputing

clusters developed in the 1990s

• Hadoop consists of:

• MapReduce

• Hadoop Distributed File System

(HDFS)


5

What’s New?


6

Characteristics of HDFS

• Very high fault tolerance

• Cannot be updated, but

corrections can be appended

• File blocks are replicated

multiple times

Three Types of Nodes:

1. Name Node (Directory)

2. Backup Node (Checkpoint)

3. Data Node (Actual Data)


7

Characteristics of MapReduce

• Map Function - Take a task and break it down into small tasks

• Reduce Function - Combine the partial answers and find the combined list

A programing framework for library and runtime (just like .NET)

• When submitting a query, this manages the Task Trackers which trigger the actual Map or Reduce task

Master (Job Tracker)

• The “doers.” Each node in the cluster has a data node and a task tracker

Workers (Task Trackers)


8

HDFS and MapReduce

The Main Node: runs the Job tracker and the name node controls the files.

Each node runs two processes: Task Tracker and Data Node


9

Basics of MapReduce

1 bill/ sec

= 400 Seconds

400 bills


10

Basics of MapReduce

1 bill/ sec

1 bill/ sec

= 200 Seconds

= 200 Seconds

200 Bills

200 BillsTotal =200 seconds


11

Basics of MapReduce

= 100 Seconds100 bills

100 bills

100 bills

100 bills

= 100 Seconds

= 100 Seconds

= 100 Seconds

Total =100 seconds


12

Basics of MapReduce

Query

Result

Name Node/Job TrackerQuery

Data Nodes/Task Trackers


13

MapReduce

• Java

• Write many lines of code

Pig

• Mostly used by Yahoo

• Most used for data processing

• Shares some constructs w/ SQL

• Is more Verbose

• Needs a lot of training for users with limited procedural programming background

• Offers control over the flow of data

Hive

• Mostly used by Facebook for analytic purposes

• Used for analytics

• Relatively easier for developers w/ SQL experience.

• Less control over optimization of data flows compared to Pig

MapReduce, Pig and Hive

• Not as efficient as MapReduce

• Higher productivity for data scientists and developers


14

Major Versions of Apache Hadoop

Apache Foundation


15

Hortonworks

• 2011: $23 million funding from Yahoo! and Benchmark Capital- positioned as an independent company

• Horton the Elephant - Horton Hears a Who!

• Employs contributors to project Apache Hadoop

• October 2011 partnered with Microsoft : Azure and Windows Server

• Cloudera founded in October 2008…started the effort to be Microsoft Azure Certified in October 2014.


16

HDP User Interface


17

HUE Architecture


Questions?Tom Marek

[email protected]

Twitter: @twmarek

Orion Gebremedhin

[email protected]

Twitter: @OrionGM

understanding apache hadoop - neudesic...• employs contributors to project apache hadoop •...

Documents