By Jeff Dean& Sanjay Ghemawat

Google Inc.OSDI 2004

Presented by : Mohit Deopujari

Map Reduce: Simplified Data Processing on Large

Clusters

2

Topics Covered in Class:

4/9/2013 Map Reduce

Cluster Architecture Distributed File Systems Coping with Node Failure Large-Scale Indexing

3

Introduction to Map-Reduce:

4/9/2013 Map Reduce

A new programming model for processing and generating large data sets A new abstraction that enables simple computations to be parallelized

while hiding away the details Map and Reduce are two primitives directly inspired by Functional

languages such as LISP This Functional model enables massive parallelization and fault-tolerance

by re-execution

4

The Programming Model:

4/9/2013 Map Reduce

Input consists of key/value pairs, Final output is also key/value pairs Computation is expressed as two functions : Map and Reduce Map :-

Written by user Input is key/value pairs, Output is intermediate key/value pairs Groups together all intermediate values associated with same

intermediate key I and passes onto Reduce Reduce :-

Also written by user Accepts intermediate key I and set of values for that key Merges together these values to form a possibly smaller set of values Zero or one output value produced per invocation An iterator is used to feed the intermediate values to the function thus

allowing us to handle lists too large to fit in memory

5

An example to count number of occurrences of each word:

4/9/2013 Map Reduce

map(String key, String value):// key: document name// value: document contentsfor each word w in value:

EmitIntermediate(w, "1");

reduce(String key, Iterator values):// key: a word// values: a list of countsint result = 0;for each v in values:

result += ParseInt(v);Emit(AsString(result));

6

An example to count number of occurrences of each word:

4/9/2013 Map Reduce

the input keys and values are drawn from a different domain than the output keys and values

the intermediate keys and values are from the same domain as the output keys and values.

7

Execution Overview:

4/9/2013 Map Reduce

1. MapReduce library in the user program splits the input file into M chunks (16 to 64MB each) and then starts up many copies of the program on a cluster of machines.

2. A special copy, the Master, assigns each of the other copies (workers) a Map or a Reduce task.

3. Each worker assigned with the Map task parses the key/value pair from the corresponding chunk of input and passes it to the Map function. Intermediate key/value pairs are produced and buffered in memory.

4. Periodically, a partitioning function partitions these pairs into R regions and writes to local disk. The locations of these pairs on disk is passed to the master.

5. When a reduce worker is notified of these locations by a master, the worker sorts the intermediate keys and groups the values by key.

8

Execution Overview:

4/9/2013 Map Reduce

6. Then, the reduce worker iterates over each unique key and passes on the list of corresponding values to the Reduce function. The output of the Reduce function for that partition is appended to the final output.

7. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user programreturns back to the user code.

9

Fault Tolerance:

4/9/2013 Map Reduce

Master Failure: Highly unlikely since there is only a single master If there is a failure, whole task is aborted and can be restarted

Worker Failure: If a Map worker fails, it is detected by Master and that node’s map tasks (even if

completed) are set to idle and made ready to be rescheduled on other machines Reduce workers are notified of the new locations from which to pick up data If a Reduce worker fails, its tasks are set to idle and made ready to be rescheduled

on other machines

10

Locality:

4/9/2013 Map Reduce

Network bandwidth is conserved because of the fact that input data resides on the local disks of the machines that make up the data cluster

The Master knows location of input files. It schedules a map task on a machine that has a copy of the corresponding input data

Thus, on large MapReduce operations, the input data is read locally and bandwidth consumption is minimal

11

Usage at Google:

4/9/2013 Map Reduce

12

Example uses :

4/9/2013 Map Reduce

distributed grep distributed sort web link-graph reversal

term-vector per host web access log stats inverted index construction

document clustering machine learning statistical machine translation

... ... ...

13

Pros :

4/9/2013 Map Reduce

Simple and powerful interface enables automatic parallelization and distribution of large scale computations

High performance achieved on large clusters of commodity CPUs (cheap hardware)

Fair fault-tolerance and capability to handle failures and still make progress Locality of input data conserves bandwidth and helps speed up tasks Wide variety of applications Was used to completely rewrite the Google Indexing code in turn making it easier

to read, manage while improving performance and efficiency

14

Cons :

4/9/2013 Map Reduce

Debugging is tricky since actual computation occurs in a distributed environment on thousand of machines with work assignments being made dynamically

In case of Map Node failure, the Map tasks need to be started all over again since the intermediate output lies in the failed machine

15

Conclusion :

4/9/2013 Map Reduce

MapReduce has been widely used at Google for many tasks primarily due to these reasons:

The model is easy to use, even for programmers without experience with parallel and distributed systems, since it hides the details of parallelization, fault-tolerance, locality optimization, and load balancing

A large variety of problems are easily expressible as MapReduce computations

Easy and efficient scalability

16

Conclusion :

4/9/2013 Map Reduce

Use of a restricted programming model (based on functional model) enables easy parallelization and fault-tolerance through re-execution

Network bandwidth is scarce hence many optimizations need to be built to minimize wastage

redundant execution can be used to reduce the impact of slow machines, and to handle machine failures and data loss