mapreduce design patterns

1© Copyright 2012 EMC Corporation. All rights reserved.

MapReduceDesign Patterns

Donald MinerGreenplum Hadoop Solutions Architect

@donaldpminer


Book was made available December 2012


Inspiration for my book


What are design patterns?(in general)

Reusable solutions to problems

Domain independent

Not a cookbook, but not a guide

Not a finished solution


Why design patterns?(in general)

Makes the intent of code easier to understand

Provides a common language for solutions

Be able to reuse code

Known performance profiles and limitations of solutions


Why MapReduce design patterns?

Recurring patterns in data-related problem solving

Groups are building patterns independently

Lots of new users every day

MapReduce is a new way of thinking

Foundation for higher-level tools (Pig, Hive, …)

Community is reaching the right level of maturity


Pattern Template

Intent

Motivation

Applicability

Structure

Consequences

Resemblances

Performance analysis

Examples


Pattern Categories

Summarization

Filtering

Data Organization

Joins

Metapatterns

Input and output


Filtering patterns Extract interesting subsets

Filtering

Bloom filtering

Top ten

Distinct

Summarization patterns top-down summaries

Numerical summarizations

Inverted index

Counting with counters

I only wantsome of my data!

I only wanta top-level view

of my data!


Data organization patterns Reorganize, restructure

Structured to hierarchical

Partitioning

Binning

Total order sorting

Shuffling

Join patterns Bringing data sets together

Reduce-side join

Replicated join

Composite join

Cartesian product

I want to changethe way my data

is organized!

I want to mashmy different datasources together!


Metapatterns Patterns of patterns

Job chaining

Chain folding

Job merging

Input and output patterns Custom input and output

Generating data

External source output

External source input

Partition pruning

I want to solvea complex problem

with multiple patterns!

I want to get data orput data in anunusual place!


Pattern: “Top Ten”(filtering)

IntentRetrieve a relatively small number of top K records, according to a ranking scheme in your data set, no matter how large the data.

MotivationFinding outliersTop ten lists are funBuilding dashboardsSorting/Limit isn’t going to work here


Pattern: “Top Ten”

Applicability Rank-able recordsLimited number of output records

ConsequencesThe top K records are returned.


Pattern: “Top Ten”Structureclass mapper: setup(): initialize top ten sorted list map(key, record): insert record into top ten sorted list if length of array is greater-than 10: truncate list to a length of 10 cleanup(): for record in top sorted ten list: emit null,record

class reducer: setup(): initialize top ten sorted list reduce(key, records): sort records truncate records to top 10 for record in records: emit record



Resemblances

SQL: SELECT * FROM table ORDER BY col4 DESC LIMIT 10;

Pig: B = ORDER A BY col4 DESC; C = LIMIT B 10;



Performance analysisPretty quick: map-heavy, low network usage

Pay attention to how many records the reducer is getting[number of input splits] x K

ExampleTop ten StackOverflow users by reputation


public static class TopTenMapper extends Mapper<Object, Text, NullWritable, Text> {

private TreeMap<Integer, Text> repToRecordMap = new TreeMap<Integer, Text>();

public void map(Object key, Text value, Context context) {

Map<String, String> parsed = MRDPUtils.transformXmlToMap(value.toString());

String userId = parsed.get("Id");

String reputation = parsed.get("Reputation");

repToRecordMap.put(Integer.parseInt(reputation), new Text(value));

if (repToRecordMap.size() > 10) {

repToRecordMap.remove(repToRecordMap.firstKey());

}

}

protected void cleanup(Context context) {

for (Text t : repToRecordMap.values()) {

context.write(NullWritable.get(), t);

}

}

}

Top Ten Mapper


public static class TopTenReducer extends Reducer<NullWritable, Text, NullWritable, Text> {

private TreeMap<Integer, Text> repToRecordMap = new TreeMap<Integer, Text>();

public void reduce(NullWritable key, Iterable<Text> values, Context context) {

for (Text value : values) {

Map<String, String> parsed = MRDPUtils.transformXmlToMap(value.toString());

repToRecordMap.put(Integer.parseInt(parsed.get("Reputation")), new Text(value));

if (repToRecordMap.size() > 10) {

repToRecordMap.remove(repToRecordMap.firstKey());

}

}

for (Text t : repToRecordMap.descendingMap().values()) {

context.write(NullWritable.get(), t);

}

}

}

Top Ten Reducer


Pattern: “Bloom Filtering”(filtering)

IntentKeep records that are a member of some predefined set of values. It is not a problem if the output is a bit inaccurate.

MotivationSimilar to normal Boolean filtering, but we are filtering on set membershipSet membership is evaluated with a Bloom filter


Pattern: “Bloom Filtering”

Applicability A feature can be extracted and tested for set membershipPredetermined set is availableSome false positives are acceptable

ConsequencesRecords that pass the Bloom filter membership test are returned

Known UsesKeep all records in a watch list (and a few records that aren’t)Pre-filtering records before an expensive membership test



Structureclass mapper: setup(): load bloom filter into memory map(key, record): if record in bloom filter:

emit (record, null)

Resemblances

UDFs?



Performance analysisMap-onlySlight overhead in moving Bloom filter into memoryBloom filter membership tests are constant time

ExampleFilter StackOverflow comments that do not contain a keywordDistributed HBase query using a Bloom filter


Candidate new patterns

Link Graph processing patterns (new category)– Shortest past, diameter, graph stats, connected

components, etc.– Too domain specific?– Has its own distinct patterns

Projection (filtering)– Remove “columns” of data

Transformation (data organization?)– Take a data set but transform it into something else


Future and call to action

Contributing your own patterns

Trends in the nature of data– Images, audio, video, biomedical, social …

Libraries, abstractions, and tools

Ecosystem patterns: YARN, HBase, ZooKeeper, …

mapreduce design patterns

Technology

emc corporation

data organization patterns

filtering patterns

recurring patterns

multiple patterns

data sets

output patterns custom

getreputation reptorecordmap