Hadoop, Hadoop, Hadoop!!!
Jerome MitchellIndiana University
Outline
• BIG DATA• Hadoop MapReduce• The Hadoop Distributed File System (HDFS)• Workflow• Conclusions• References• HandsOn
LOTS OF DATA EVERYWHERE
Why Should You Care? Even Grocery Stores Care
What is Hadoop?
• At Google MapReduce operation are run on a special file system called Google File System (GFS) that is highly optimized for this purpose.
• GFS is not open source.• Doug Cutting and Yahoo! reverse engineered the GFS
and called it Hadoop Distributed File System (HDFS).• The software framework that supports HDFS,
MapReduce and other related entities is called the project Hadoop or simply Hadoop.
• This is open source and distributed by Apache.
What exactly is Hadoop
• A growing collection of subprojects
Motivation for MapReduce
• Large-Scale Data Processing– Want to use 1000s of CPUs
• But, don’t want the hassle of managing things
• MapReduce Architecture provides– Automatic Parallelization and Distribution – Fault Tolerance– I/O Scheduling– Monitoring and Status Updates
9
MapReduce Model• Input & Output: a set of key/value pairs• Two primitive operations
– map: (k1,v1) list(k2,v2)
– reduce: (k2,list(v2)) list(k3,v3)
• Each map operation processes one input key/value pair and produces a set of key/value pairs
• Each reduce operation– Merges all intermediate values (produced by map ops) for a particular key– Produce final key/value pairs
• Operations are organized into tasks– Map tasks: apply map operation to a set of key/value pairs– Reduce tasks: apply reduce operation to intermediate key/value pairs– Each MapReduce job comprises a set of map and reduce (optional) tasks.
HDFS Architecture
The WorkFlow
• Load data into the Cluster (HDFS writes)• Analyze the data (MapReduce)• Store results in the Cluster (HDFS)• Read the results from the Cluster (HDFS reads)
Distributed Data Processing
Slaves
Master
Client
Distributed Data Storage
Name NodeSecondary Name
Node
Data Nodes &Task Tracker
Data Nodes &Task Tracker
Data Nodes &Task Tracker
Data Nodes &Task Tracker
Data Nodes &Task Tracker
Data Nodes &Task Tracker
MapReduce HDFS
Job Tracker
Hadoop Server Roles
Hadoop Cluster
Switch Switch
Switch Switch Switch
DN + TTDN + TTDN + TTDN + TT
DN + TTDN + TTDN + TTDN + TT
DN + TTDN + TTDN + TTDN + TT
DN + TTDN + TTDN + TTDN + TTDN + TT
Switch
Rack 1 Rack 2 Rack 3 Rack N
Hadoop Rack Awareness – Why?
Data Node 1
Data Node 2
Data Node 3Data Node 5
Rack 1 Rack 5 Rack 9
Switch Switch Switch
Data Node 5
Data Node 6
Data Node 7Data Node 8
Data Node 5
Data Node 10
Data Node 11
Data Node 12
Switch
Name Node
Results.txt =BLK A:DN1, DN5, DN6
BLK B:DN7, DN1, DN2
BLK CDN5, DN8, DN9
metatadata
Rack 1:Data Node 1
Data Node 2Data Node 3
Rack 1:Data Node 5Data Node 6Data Node 7
Rack Aware
Sample Scenario
Huge file containing all emails sent to Indiana University… File.txt
How many times did our customers type the word refund into emails sent to Indiana University?
Writing Files to HDFSFile.txt
Data Node 1 Data Node 5 Data Node 6
BLK A BLK B BLK C
BLK A BLK B BLK C
Data Node N
Client
NameNode
Data Node Reading Files From HDFS
Data Node 1
Data Node 2
Data NodeData NodeRack 1 Rack 5 Rack 9
Switch Switch Switch
Data Node 1
Data Node 2
Data NodeData Node
Data Node 1
Data Node 2
Data NodeData Node
Switch
Name Node
Results.txt =BLK A:DN1, DN5, DN6
BLK B:DN7, DN1, DN2
BLK CDN5, DN8, DN9
metatadata
Rack 1:Data Node 1 Data Node 2Data Node 3
Rack 1:Data Node 5
Rack Aware
MapReduce: Three Phases
1. Map2. Sort
3. Reduce
Data Processing: Map
Map Task Map Task Map Task
BLK A BLK B BLK C
Job TrackerName Node
Data Node 1 Data Node 5 Data Node 9
File.txt
MapReduce: The Map Step
vk
k v
k v
mapvk
vk
…
k vmap
Inputkey-value pairs
Intermediatekey-value pairs
…
k v
Data Processing: Reduce
Map Task Map Task Map Task
Reduce TaskResults.txt
HDFS
BLK A BLK B BLK C
MapReduce: The Reduce Step
k v
…
k v
k v
k v
Intermediatekey-value pairs
group
reduce
reduce
k v
k v
k v
…
k v
…
k v
k v v
v v
Key-value groupsOutput key-value pairs
Clients Reading Files from HDFS
Data Node 1
Data Node 2
Data NodeData NodeRack 1 Rack 5 Rack 9
Switch Switch Switch
Data Node 1
Data Node 2
Data NodeData Node
Data Node 1
Data Node 2
Data NodeData Node
Client
NameNode
Results.txt =BLK A:DN1, DN5, DN6
BLK B:DN7, DN1, DN2
BLK CDN5, DN8, DN9
metatadata
Conclusions
• We introduced MapReduce programming model for processing large scale data
• We discussed the supporting Hadoop Distributed File System
• The concepts were illustrated using a simple example
• We reviewed some important parts of the source code for the example.
References
1. Apache Hadoop Tutorial: http://hadoop.apache.org http://hadoop.apache.org/core/docs/current/mapred_tutorial.html
2. Dean, J. and Ghemawat, S. 2008. MapReduce: simplified data processing on large clusters. Communication of ACM 51, 1 (Jan. 2008), 107-113.
3. Cloudera Videos by Aaron Kimball: http://www.cloudera.com/hadoop-training-basic4.
http://www.cse.buffalo.edu/faculty/bina/mapreduce.html
FINALLY, A HANDS-ON ASSIGNMENT!
The MapReduce Framework (pioneered by Google)
Automatic Parallel Execution in MapReduce (Google)
Handles failures automatically, e.g., restarts tasks if a node fails; runs multiples copies of the same task to
avoid a slow task slowing down the whole job
The Map (Example)
When in the course of human events it …
It was the best of times and the worst of times…
map(in,1) (the,1) (of,1) (it,1) (it,1) (was,1) (the,1) (of,1) …
(when,1), (course,1) (human,1) (events,1) (best,1) …
inputs tasks (M=3) partitions (intermediate files) (R=2)
This paper evaluates the suitability of the … map (this,1) (paper,1) (evaluates,1) (suitability,1) …
(the,1) (of,1) (the,1) …
Over the past five years, the authors and many…
map (over,1), (past,1) (five,1) (years,1) (authors,1) (many,1) …
(the,1), (the,1) (and,1) …
The Reduce (Example)
reduce
(in,1) (the,1) (of,1) (it,1) (it,1) (was,1) (the,1) (of,1) …
(the,1) (of,1) (the,1) …
reduce taskpartition (intermediate files) (R=2)
(the,1), (the,1) (and,1) …
sort
(and, (1)) (in,(1)) (it, (1,1)) (the, (1,1,1,1,1,1)) (of, (1,1,1)) (was,(1))
(and,1) (in,1) (it, 2) (of, 3) (the,6) (was,1)
Note: only one of the two reduce tasks shown
Lifecycle of a MapReduce Job
Map function
Reduce function
Run this program as aMapReduce job
Job Configuration Parameters• 190+ parameters in
Hadoop• Set manually or defaults
are used
Formatting the NameNode
Before we start, we have to format Hadoop's distributed filesystem (HDFS) for the namenode. You need to do this the first time you set up Hadoop. Do not format a running Hadoop namenode, this will cause all your data in the HDFS filesytem to be erased.
To format the filesystem, run the command (from the master):
--------------------------------------------
bin/hadoop namenode -format
---------------------------------------------
Starting Hadoop:
Starting hadoop is done in two steps: First, the HDFS daemons are started: the namenode daemon is started on master, and datanode daemons are started on all slaves (here: master and slave). Second, the MapReduce daemons are started: the jobtracker is started on master, and tasktracker daemons are started on all slaves (here: master and slave).
HDFS daemons:Run the command <HADOOP_HOME>/bin/start-dfs.sh on the machine you want the namenode to run on. This will bring up HDFS with the namenode running on the machine you ran the previous command on, and datanodes on the machines listed in the conf/slaves file. In our case, we will run bin/start-dfs.sh on master: -------------------------bin/start-dfs.sh------------------------On slave, you can examine the success or failure of this command by inspecting the log file <HADOOP_HOME>/logs/hadoop-hadoop-datanode-slave.log.
Now, the following Java processes should run on master:root@ubuntu:$HADOOP_HOME/bin: jps 14799 NameNode 15314 Jps 14880 DataNode 14977 SecondaryNameNode ------------------------------------
MapReduce Daemons:
In our case, we will run bin/start-mapred.sh on master:
-------------------------------------
bin/start-mapred.sh
-------------------------------------
On slave, you can examine the success or failure of this command by inspecting the log file <HADOOP_HOME>/logs/hadoop-hadoop-tasktracker-slave.log.
At this point, the following Java processes should run on master:
----------------------------------------------------
root@ubuntu:$HADOOP_HOME/bin$ jps
16017 Jps
14799 NameNode
15686 TaskTracker
14880 DataNode
15596 JobTracker
14977 SecondaryNameNode
----------------------------------------------------
We will execute your first Hadoop MapReduce job. We will use the WordCount example, which will read a text file and count the frequency of words. The input is a text file and the output is a text file with each line of which contains a word and the count of how often it occurred, separated by a tab. Download example input data:
Copy local data file to HDFSBefore we run the actual MapReduce job, we first have to copy the files from our local file system to Hadoop's HDFS-----------------------------root@ubuntu:$HADOOP_HOME/bin$ hadoop dfs -copyFromLocal /tmp/source destination
Build WordCountExecute build.sh
Run the MapReduce jobNow, we actually run the WordCount example job. This command will read all the files in the HDFS “destination” directory , process it, and store the result in the HDFS directory “output”.-----------------------------------------root@ubuntu:$HADOOP_HOME/bin bin/hadoop hadoop-example wordcount destination output-----------------------------------------You can check if the result is successfully stored in HDFS directory “output”.
Retrieve the job result from HDFSTo inspect the file, you can copy it from HDFS to the local file system.-------------------------------------root@ubuntu:/usr/local/hadoop$ mkdir /tmp/outputroot@ubuntu:/usr/local/hadoop$ bin/hadoop dfs –copyToLocal output/part-00000 /tmp/output ---------------------------------------- Alternatively, you can read the file directly from HDFS without copying it to the local file system by using the command :---------------------------------------------root@ubuntu:/usr/local/hadoop$ bin/hadoop dfs –cat output/part-00000
Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++. Creating a launching program for your applicationThe launching program configures: – The Mapper and Reducer to use – The output key and value types (input types are inferred from the InputFormat) – The locations for your input and outputThe launching program then submits the job and typically waits for it to complete
A Map/Reduce may specify how it’s input is to be read by specifying an InputFormat to be used A Map/Reduce may specify how it’s output is to be written by specifying an OutputFormat to be used