map reduce for data-intensive computing
DESCRIPTION
Map Reduce for data-intensive computing. (Some of the content is adapted from the original authors’ talk at OSDI 04). Motivation. Google needed to process web-scale data Data much larger than what fits on one machine Need parallel processing to get results in a reasonable time - PowerPoint PPT PresentationTRANSCRIPT
Map Reduce for data-intensive computing
(Some of the content is adapted from the original authors’ talk at OSDI 04)
Motivation
• Google needed to process web-scale data– Data much larger than what fits on one machine– Need parallel processing to get results in a
reasonable time– Use cheap commodity machines to do the job
Requirements
• Solution must– Scale to 1000s of compute nodes– Must automatically handle faults– Provide monitoring of jobs– Be easy for programmers to use
MapReduce Programming model
• Input and Output are sets of key/value pairs• Programmer provides two functions– map(K1, V1) -> list(K2, V2)• Produces list of intermediate key/value pairs for each
input key/value pair– reduce(K2, list(V2)) -> list(K3, V3)• Produces a list of result values for all intermediate
values that are associated with the same intermediate key
MapReduce Pipeline
Map Shuffle Reduce
Read from DFS
Write to DFS
MapReduce in Action
Map (k1, v1) list(k2, v2)• Processes one input key/value pair • Produces a set of intermediate key/value pairs
Reduce (k2, list(v2)) list(k3, v3) • Combines intermediate values for one particular key • Produces a set of merged output values (usually one)
MapReduce Architecture
MapReduce MapReduce MapReduce MapReduce
Distributed File System
Network
MapReduceJob Tracker
Software Components
• Job Tracker (Master)– Maintains Cluster membership of workers– Accepts MR jobs from clients and dispatches tasks to
workers– Monitors workers’ progress– Restarts tasks in the event of failure
• Task Tracker (Worker)– Provides an environment to run a task– Maintains and serves intermediate files between Map
and Reduce phases
MapReduce Parallelism
Hash Partitioning
Example 1: Count Word Occurrences
• Input: Set of (Document name, Document Contents)• Output: Set of (Word, Count(Word))• map(k1, v1):
for each word w in v1emit(w, 1)
• reduce(k2, v2_list):int result = 0;for each v in v2_list
result += v;emit(k2, result)
Map
Example 1: Count Word Occurrences
Map
Reduce
Reduce
this is a line
this is another line
another line
yet another line
this, 1is, 1a, 1line, 1this, 1is, 1another, 1line, 1
another, 1line, 1
yet, 1another, 1line, 1
a, 1another, 1is, 1is, 1
line, 1line, 1this, 1this, 1
another, 1another, 1
line, 1line, 1yet, 1
a, 1another, 3is, 2
line, 4this, 2yet, 1
Example 2: Joins
• Input: Rows of Relation R, Rows of Relation S• Output: R join S on R.x = S.y• map(k1, v1)
if (input == R)emit(v1.x, [“R”, v1])
elseemit(v1.y, [“S”, v2])
• reduce(k2, v2_list)for r in v2_list where r[1] == “R”
for s in v2_list where s[1] == “S”emit(1, result(r[2], s[2]))
Other examples
• Distributed grep• Inverted index construction• Machine learning• Distributed sort• …
Fault Tolerant Evaluation
• Task Fault Tolerance achieved through re-execution
• All consumers consume data only after completely generated by the producer– This is an important property to isolate faults to
one task• Task completion committed through Master• Cannot handle master failure
Task granularity and pipelining
• Fine granularity tasks– Many more map tasks than machines• Minimizes time for fault recovery• Pipelines shuffling with map execution• Better load balancing
Optimization: Combiners
• Sometimes partial aggregation is possible on the Map side
• May cut down amount of data to be transferred to the reducer (sometimes significantly)
• combine(K2, list(V2)) -> K2, list(V2)• For Word Occurrence Count example,
Combine == Reduce
Map
Example 1: Count Word Occurrences (With Combiners)
Map
Reduce
Reduce
this is a line
this is another line
another line
yet another line
this, 1is, 1a, 1line, 1this, 1is, 1another, 1line, 1
another, 1line, 1
yet, 1another, 1line, 1
a, 1another, 1is, 2
line, 2this, 2
another, 2
line, 2yet, 1
a, 1another, 3is, 2
line, 4this, 2yet, 1
Optimization: Redundant Execution
• Slow workers lengthen completion time• Slowness happens because of– Other jobs consuming resources– Bad disks/network etc
• Solution: Near the end of the job spawn extra copies of long running tasks– Whichever copy finishes first, wins.– Kill the rest
• In Hadoop this is called “speculative execution”
Optimization: Locality Optimization
• Task scheduling policy– Ask DFS for locations of replicas of input file blocks– Map tasks scheduled so that input blocks are
machine local or rack local• Effect: Tasks read data at local disk speeds• Without this rack switches limit data rate
Distributed Filesystem
• Used as the “store” of data• MapReduce reads input from DFS and writes
output to DFS• Provides a “shared disk” view to applications
using local storage on shared-nothing hardware
• Provides redundancy by replication to protect from node/disk failures
DFS Architecture
Taken from Ghemawat’s SOSP’03 paper (The Google Filesystem)
• Single Master (with backups) that track DFS file name to chunk mapping• Several Chunk servers that store chunks on local disks• Chunk Size ~ 64MB or larger• Chunks are replicated• Master only used for chunk lookups – Does not participate in transfer of data
Chunk Replication
• Several Replicas of each Chunk– Usually 3
• Replicas usually spread across racks and data centers to maximize availability
• Master tracks location of each replica of a chunk• When chunk failure is detected, master automatically
rebuilds new replica to maintain replication level• Automatically picks chunk servers for new replicas
based on utilization
Offline Reading
• Original MapReduce Paper– “Simplified Data Processing on Large Clusters” by
Jeffrey Dean and Sanjay Ghemawat in OSDI’04• Original DFS Paper– “The Google Filesystem” by Sanjay Ghemawat,
Howard Gobioff, and Shun-Tak Leung in SOSP ‘03• CACM Papers on MapReduce and Parallel DBMSs– MapReduce and Parallel DBMSs: Friends or Foes?– MapReduce: A Flexible Data Processing Tool