MapReduce: MapReduce: simplified data simplified data
processing on large processing on large clustersclustersJeffrey Dean and Jeffrey Dean and
Sanjay GhemawatSanjay Ghemawat
Presented By :-Presented By :-Venkataramana ChunduruVenkataramana Chunduru
AGENDAAGENDA GFSGFS MAP REDUCEMAP REDUCE HADOOPHADOOP
MotivationMotivation Input data is large.Input data is large.
The whole Web, billions of Pages.The whole Web, billions of Pages. Lots of machinesLots of machines
Use them efficiently.Use them efficiently.
Google needed good Distributed file Google needed good Distributed file SystemSystem
Why not use the existing file systems?Why not use the existing file systems? Google’s problems are different from anyone else.Google’s problems are different from anyone else. GFS is designed for Google apps and workloads.GFS is designed for Google apps and workloads. Google apps are designed for GFS.Google apps are designed for GFS.
NFS DisadvantagesNFS Disadvantages Network congestion Heavy disk activity of the NFS server adversely
affects the NFS’s performance. When the client attempts to mount , the client
system hangs, although this can be mitigated using a specific mount.
If the server hosting the exportedfile system becomes unavailable due to any reason, no one can access the
resource. NFS has security problems because its design assumes a trustednetwork.
GFS AssumptionsGFS Assumptions High Component failure ratesHigh Component failure rates
Inexpensive commodity components fail all the Inexpensive commodity components fail all the time.time.
Modest number of huge filesModest number of huge files.. Just a few millionJust a few million Each is 100 MB or larger: multi GB files Each is 100 MB or larger: multi GB files
typicallytypically Files are write once ,mostly appended Files are write once ,mostly appended
toto Perhaps ConcurrentlyPerhaps Concurrently
Large streaming readsLarge streaming reads..
GFS Design DecisionsGFS Design Decisions Files are stored as chunks.Files are stored as chunks. - Fixed size(64 MB).- Fixed size(64 MB). Reliability through replication.Reliability through replication. - - Each chunk is replicated across 3+ chunkserversEach chunk is replicated across 3+ chunkservers Single master to co ordinate Single master to co ordinate
access,keep metadataaccess,keep metadata - - Simple centralized management.Simple centralized management. No data cachingNo data caching - - Little benefit due to large datasets,streaming reads.Little benefit due to large datasets,streaming reads.
GFS ArchitectureGFS Architecture
Single MasterSingle Master From Distributed systems we know it From Distributed systems we know it
is a :is a : - Single point of failure.- Single point of failure. - Scalibility bottleneck.- Scalibility bottleneck. GFS solutionsGFS solutions - Shadow masters- Shadow masters - Minimize master involvement- Minimize master involvement Simple and good enoughSimple and good enough..
Metadata (1/2)Metadata (1/2) Global metadata is stored on the Global metadata is stored on the
master.master. - File and chunk namespaces.- File and chunk namespaces. - Mapping from files to chunks.- Mapping from files to chunks. - Locations of each chunk replicas.- Locations of each chunk replicas. All in memory (64bytes/chunk)All in memory (64bytes/chunk) - Fast- Fast - Easily Accessible.- Easily Accessible.
Metadata (2/2)Metadata (2/2) Master has an operation log for Master has an operation log for
persistent logging of critical metadata persistent logging of critical metadata updates.updates.
- Persistent on local disk- Persistent on local disk - Replicated- Replicated - Check points for faster recovery.- Check points for faster recovery.
Deployment in GoogleDeployment in Google 50 + GFS Clusters50 + GFS Clusters
Each with thousands of storage nodesEach with thousands of storage nodes
Managing petabytes of data.Managing petabytes of data.
GFS is under big table.GFS is under big table.
Conclusion of GFSConclusion of GFS GFS demonstrates how to support large GFS demonstrates how to support large
scale processing workloads on scale processing workloads on commodity hardwarecommodity hardware
- Designed to tolerate frequent component failures.- Designed to tolerate frequent component failures. - Optimized for huge files that are mostly appended - Optimized for huge files that are mostly appended
and read.and read. - Go for simple solutions.- Go for simple solutions.
GFS has met Google's storage needs…. GFS has met Google's storage needs…. it must be good !!!it must be good !!!
Example for MapReduceExample for MapReduce Page 1: the weather is goodPage 1: the weather is good Page 2: today is goodPage 2: today is good Page 3: good weather is good.Page 3: good weather is good.
Map outputMap output Worker 1:Worker 1:
(the 1), (weather 1), (is 1), (good 1).(the 1), (weather 1), (is 1), (good 1). Worker 2:Worker 2:
(today 1), (is 1), (good 1).(today 1), (is 1), (good 1). Worker 3:Worker 3:
(good 1), (weather 1), (is 1), (good 1).(good 1), (weather 1), (is 1), (good 1).
Reduce InputReduce Input Worker 1:Worker 1:
(the 1)(the 1) Worker 2:Worker 2:
(is 1), (is 1), (is 1)(is 1), (is 1), (is 1) Worker 3:Worker 3:
(weather 1), (weather 1)(weather 1), (weather 1) Worker 4:Worker 4:
(today 1)(today 1) Worker 5:Worker 5:
(good 1), (good 1), (good 1), (good 1)(good 1), (good 1), (good 1), (good 1)
Reduce OutputReduce Output Worker 1:Worker 1:
(the 1)(the 1) Worker 2:Worker 2:
(is 3)(is 3) Worker 3:Worker 3:
(weather 2)(weather 2) Worker 4:Worker 4:
(today 1)(today 1) Worker 5:Worker 5:
(good 4)(good 4)
MapReduce ArchitectureMapReduce Architecture
Parallel ExecutionParallel Execution
Fault ToleranceFault Tolerance Network Failure:Network Failure:
Detect failure via periodic heartbeats Detect failure via periodic heartbeats Re-execute completed and in-progress Re-execute completed and in-progress mapmap
tasks tasks Re-execute in progress Re-execute in progress reducereduce tasks tasks Task completion committed through masterTask completion committed through master
Master failure:Master failure: Could handle, but don't yet (master failure Could handle, but don't yet (master failure
unlikely)unlikely)
RefinementRefinement
Different partitioning functions.Different partitioning functions. Combiner function.Combiner function. Different input/output types.Different input/output types. Skipping bad records.Skipping bad records. Local execution.Local execution. Status info.Status info. Counters.Counters.
What’sWhat’s Framework for running applications on large clusters of Framework for running applications on large clusters of
commodity hardwarecommodity hardware Scale: petabytes of data on thousands of nodesScale: petabytes of data on thousands of nodes
IncludeInclude Storage: HDFSStorage: HDFS Processing: MapReduceProcessing: MapReduce
Support the Map/Reduce programming modelSupport the Map/Reduce programming model Requirements Requirements
Economy: use cluster of comodity computersEconomy: use cluster of comodity computers EEasy to useasy to use
Users: no need to deal with the complexity of Users: no need to deal with the complexity of distributed computingdistributed computing
Reliable: can handle node failures automaticallyReliable: can handle node failures automatically
Whats Hadoop ..Contd.Whats Hadoop ..Contd. Hadoop is a software platform that lets one Hadoop is a software platform that lets one
easily write and run applications that easily write and run applications that process vast amounts of data.process vast amounts of data.
Here's what makes Hadoop especially useful:Here's what makes Hadoop especially useful:
ScalableScalable
EconomicalEconomical
EfficientEfficient
Reliable Reliable
HDFSHDFS Hadoop implements Hadoop implements MapReduceMapReduce, using the , using the
Hadoop Distributed File System (HDFS) (see Hadoop Distributed File System (HDFS) (see figure below.) figure below.)
MapReduce divides applications into many MapReduce divides applications into many small blocks of work. HDFS creates multiple small blocks of work. HDFS creates multiple replicas of data blocks for reliability, placing replicas of data blocks for reliability, placing them on compute nodes around the cluster. them on compute nodes around the cluster. MapReduce can then process the data where MapReduce can then process the data where it is located. it is located.
Hadoop has been demonstrated on clusters Hadoop has been demonstrated on clusters with 2000 nodes. The current design target is with 2000 nodes. The current design target is 10,000 node clusters.10,000 node clusters.
Hadoop ArchitectureHadoop Architecture
DataData data data data dataData data data data dataData data data data data
Data data data data dataData data data data dataData data data data data
Data data data data dataData data data data dataData data data data data
Data data data data dataData data data data dataData data data data data
ResultsData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data dataData data data data
Hadoop Cluster
DFS Block 1
DFS Block 1
DFS Block 2
DFS Block 2
DFS Block 2
DFS Block 1
DFS Block 3
DFS Block 3
DFS Block 3
MAP
MAP
MAP
Reduce
Sample Hadoop CodeSample Hadoop Code Sample text-files as input:Sample text-files as input: $ bin/hadoop dfs -ls /usr/joe/wordcount/input/ $ bin/hadoop dfs -ls /usr/joe/wordcount/input/
/usr/joe/wordcount/input/file01 /usr/joe/wordcount/input/file01 /usr/joe/wordcount/input/file02 /usr/joe/wordcount/input/file02
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file01file01 Hello World, Bye World! Hello World, Bye World!
$ bin/hadoop dfs -cat /usr/joe/wordcount/input/$ bin/hadoop dfs -cat /usr/joe/wordcount/input/file02file02 Hello Hadoop, Goodbye to hadoop. Hello Hadoop, Goodbye to hadoop.
Run the application:Run the application: $ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount $ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount
/usr/joe/wordcount/input /usr/joe/wordcount/output /usr/joe/wordcount/input /usr/joe/wordcount/output Output:Output: $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000 $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1 Bye 1 Goodbye 1 Goodbye 1 Hadoop, 1 Hadoop, 1 Hello 2 Hello 2 World! 1 World! 1 World, 1 World, 1 hadoop. 1 hadoop. 1 to 1 to 1
Contd…Contd… Notice that the inputs differ from the first version we looked at, and how Notice that the inputs differ from the first version we looked at, and how
they affect the outputs.they affect the outputs. Now, lets plug-in a pattern-file which lists the word-patterns to be Now, lets plug-in a pattern-file which lists the word-patterns to be
ignored, via the DistributedCache.ignored, via the DistributedCache. $ hadoop dfs -cat /user/joe/wordcount/$ hadoop dfs -cat /user/joe/wordcount/patterns.txt patterns.txt
\. \. \, \, \! \! to to
Run it again, this time with more options:Run it again, this time with more options: $ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount -$ bin/hadoop jar /usr/joe/wordcount.jar org.myorg.WordCount -
Dwordcount.Dwordcount.case.sensitive=true case.sensitive=true /usr/joe/wordcount/input /usr/joe/wordcount/input /usr/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt /usr/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt
As expected, the output:As expected, the output: $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000 $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-00000
Bye 1 Bye 1 Goodbye 1 Goodbye 1 Hadoop 1 Hadoop 1 Hello 2 Hello 2 World 2 World 2 hadoop 1 hadoop 1
Contd…Contd… Run it once more, this time switch-off case-sensitivity:Run it once more, this time switch-off case-sensitivity: $ bin/hadoop jar /usr/joe/wordcount.jar $ bin/hadoop jar /usr/joe/wordcount.jar
org.myorg.WordCount -org.myorg.WordCount -Dwordcount.Dwordcount.case.sensitive=falsecase.sensitive=false /usr/joe/wordcount/input /usr/joe/wordcount/output -/usr/joe/wordcount/input /usr/joe/wordcount/output -skip /user/joe/wordcount/patterns.txt skip /user/joe/wordcount/patterns.txt
Sure enough, the output:Sure enough, the output: $ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-$ bin/hadoop dfs -cat /usr/joe/wordcount/output/part-
00000 00000 bye 1 bye 1 goodbye 1 goodbye 1 hadoop 2 hadoop 2 hello 2 hello 2 world 2world 2
HadoopHadoop HDFS assumes that hardware is unreliable HDFS assumes that hardware is unreliable
and will eventually fail.and will eventually fail.
Similar to RAID level exceptSimilar to RAID level except --HDFS can replicate data across several HDFS can replicate data across several
machinesmachines
Provides Fault toleranceProvides Fault tolerance
Extremely high capacity storageExtremely high capacity storage
HadoopHadoop
““Moving Computation is cheaper than Moving Computation is cheaper than moving data”moving data”
HDFS is said to be rack awareHDFS is said to be rack aware..
Who uses Hadoop?Who uses Hadoop? ‘‘Facebook’Facebook’ uses Hadoop to analyze user uses Hadoop to analyze user
behavior and the effectiveness of ads on behavior and the effectiveness of ads on the site.the site.
The tech team at ‘The tech team at ‘The New York Times’The New York Times’
rented computing power on ‘rented computing power on ‘Amazon’s’Amazon’s’ cloud and used Hadoop to convert 11 cloud and used Hadoop to convert 11 million archived articles, dating back to million archived articles, dating back to 1851, to digital and searchable documents. 1851, to digital and searchable documents. They turned around in a single day a job They turned around in a single day a job that otherwise would have taken months.”that otherwise would have taken months.”
Who uses Hadoop?Who uses Hadoop? Besides Besides Yahoo!,Yahoo!, many other organizations are using many other organizations are using
Hadoop to run large distributed computations. Some of Hadoop to run large distributed computations. Some of them include:them include:
A9.comA9.com FacebookFacebook Fox Interactive MediaFox Interactive Media IBMIBM ImageShackImageShack ISIISI Joost Joost Last.fm Last.fm Powerset Powerset The New York Times The New York Times Rackspace Rackspace Veoh Veoh
Yahoo! Launches World's Yahoo! Launches World's Largest Hadoop Production Largest Hadoop Production
ApplicationApplication YAHOO! RECENTLY LAUNCHED WHAT WE BELIEVE IS YAHOO! RECENTLY LAUNCHED WHAT WE BELIEVE IS
THE WORLDS LARGEST THE WORLDS LARGEST APACHE HADOOPAPACHE HADOOP PRODUCTION PRODUCTION APPLICATION. THE YAHOO! SEARCH WEBMAP IS A APPLICATION. THE YAHOO! SEARCH WEBMAP IS A HADOOP APPLICATION THAT RUNS ON A MORE THAN HADOOP APPLICATION THAT RUNS ON A MORE THAN 10,000 CORE LINUX CLUSTER AND PRODUCES DATA 10,000 CORE LINUX CLUSTER AND PRODUCES DATA THAT IS NOW USED IN EVERY YAHOO! WEB SEARCH THAT IS NOW USED IN EVERY YAHOO! WEB SEARCH QUERY.QUERY.
THE WEBMAP BUILD STARTS WITH EVERY WEB PAGE THE WEBMAP BUILD STARTS WITH EVERY WEB PAGE CRAWLED BY YAHOO! AND PRODUCES A DATABASE OF CRAWLED BY YAHOO! AND PRODUCES A DATABASE OF ALL KNOWN WEB PAGES AND SITES ON THE INTERNET ALL KNOWN WEB PAGES AND SITES ON THE INTERNET AND A VAST ARRAY OF DATA ABOUT EVERY PAGE AND AND A VAST ARRAY OF DATA ABOUT EVERY PAGE AND SITE. THIS DERIVED DATA FEEDS THE MACHINE SITE. THIS DERIVED DATA FEEDS THE MACHINE LEARNED RANKING ALGORITHMS AT THE HEART OF LEARNED RANKING ALGORITHMS AT THE HEART OF YAHOO! SEARCH.YAHOO! SEARCH.
Yahoo’s HadoopYahoo’s Hadoop One of Yahoo's One of Yahoo's HadoopHadoop clusters sorted 1 terabyte of data in 209 clusters sorted 1 terabyte of data in 209
seconds, which beat the previous record of 297 seconds in the annual seconds, which beat the previous record of 297 seconds in the annual general purpose (daytona) general purpose (daytona) terabyte sort benchmarkterabyte sort benchmark. The sort . The sort benchmark, which was created in 1998 by Jim Gray, specifies the benchmark, which was created in 1998 by Jim Gray, specifies the input data (10 billion 100 byte records), which must be completely input data (10 billion 100 byte records), which must be completely sorted and written to disk. This is the first time that either a Java or sorted and written to disk. This is the first time that either a Java or an open source program has won. Yahoo is both the largest user of an open source program has won. Yahoo is both the largest user of Hadoop with 13,000+ nodes running hundreds of thousands of jobs a Hadoop with 13,000+ nodes running hundreds of thousands of jobs a month and the largest contributor, although non-Yahoo month and the largest contributor, although non-Yahoo usageusage and and contributionscontributions are are increasing rapidly increasing rapidly..
The cluster statistics were:The cluster statistics were:
910 nodes, 2 quad core Xeons @ 2.0ghz per node910 nodes, 2 quad core Xeons @ 2.0ghz per node
4 SATA disks per node, 8G RAM per node4 SATA disks per node, 8G RAM per node
1 gigabit ethernet on each node, 40 nodes per rack1 gigabit ethernet on each node, 40 nodes per rack
8 gigabit ethernet uplinks from each rack to the core.8 gigabit ethernet uplinks from each rack to the core.
Red Hat Enterprise Linux Server Release 5.1 (kernel 2.6.18)Red Hat Enterprise Linux Server Release 5.1 (kernel 2.6.18)
Sun Java JDK 1.6.0_05-b13Sun Java JDK 1.6.0_05-b13
Process DiagramProcess Diagram
Map/Reduce ProcessesMap/Reduce Processes
• • Launching ApplicationLaunching Application – – User application codeUser application code – – Submits a specific kind of Map/Reduce jobSubmits a specific kind of Map/Reduce job• • JobTrackerJobTracker – – Handles all jobsHandles all jobs – – Makes all scheduling decisionsMakes all scheduling decisions• • TaskTrackerTaskTracker – – Manager for all tasks on a given nodeManager for all tasks on a given node• • TaskTask – – Runs an individual map or reduce fragment for aRuns an individual map or reduce fragment for a given jobgiven job – – Forks from the TaskTrackerForks from the TaskTracker
Hadoop Map-Reduce Hadoop Map-Reduce ArchitectureArchitecture
Master-Slave architectureMaster-Slave architecture
Map-Reduce Master “Jobtracker”Map-Reduce Master “Jobtracker”
– – Accepts MR jobs submitted by usersAccepts MR jobs submitted by users – – Assigns Map and Reduce tasks to TasktrackersAssigns Map and Reduce tasks to Tasktrackers – – Monitors task and tasktracker status, re-executes tasks Monitors task and tasktracker status, re-executes tasks
upon upon failurefailure
Map-Reduce Slaves “Tasktrackers”Map-Reduce Slaves “Tasktrackers”
– – Run Map and Reduce tasks upon instruction from the Run Map and Reduce tasks upon instruction from the JobtrackerJobtracker
– – Manage storage and transmission of intermediate outputManage storage and transmission of intermediate output
Imp LinksImp Links http://public.yahoo.com/gogate/http://public.yahoo.com/gogate/
hadoop-tutorial/start-tutorial.htmlhadoop-tutorial/start-tutorial.html http://www.youtube.com/watch?http://www.youtube.com/watch?
v=5Eib_H_zCEY&feature=relatedv=5Eib_H_zCEY&feature=related http://www.youtube.com/watch?http://www.youtube.com/watch?
v=yjPBkvYh-ss&feature=relatedv=yjPBkvYh-ss&feature=related http://labs.google.com/papers/gfs-http://labs.google.com/papers/gfs-
sosp2003.pdfsosp2003.pdf
Thank you !!!!!Thank you !!!!!