embracing the data deluge: data-intensive computing for the masses
DESCRIPTION
Embracing the Data Deluge: Data-Intensive Computing for the Masses. Jimmy Lin University of Maryland Tuesday, July 13, 2010. - PowerPoint PPT PresentationTRANSCRIPT
Embracing the Data Deluge:Data-Intensive Computing for the Masses
Jimmy LinUniversity of Maryland
Tuesday, July 13, 2010
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United StatesSee http://creativecommons.org/licenses/by-nc-sa/3.0/us/ for details
Introduction We live in a world of large data…
Staying relevant requires embracing it! In text processing…
Emergence and dominance of empirical, data-driven research Constant danger: uninteresting conclusions on “toy” datasets
(or, experiments taking forever) In the natural sciences…
Emergence of the 4th Paradigm: data-intensive eScience Difficult computer science problems!
How do we practically scale to large datasets? Case study in text processing: statistical machine translation Case study in bioinformatics: DNA sequence alignment
How much data? Google processes 20 PB a day (2008) Wayback Machine has 3 PB + 100 TB/month (3/2009) eBay has 6.5 PB of user data + 50 TB/day (5/2009) Facebook has 36 PB of user data + 80-90 TB/day (6/2010) CERN’s LHC: 15 PB a year (any day now) LSST: 6-10 PB a year (~2015)
640K ought to be enough for anybody.
No data like more data!
(Banko and Brill, ACL 2001)(Brants et al., EMNLP 2007)
s/knowledge/data/g;
How do we get here if we’re not Google?
+ simple, distributed programming models cheap commodity clusters
= data-intensive computing for the masses!
(or utility computing)
Source: flickr (turtlemom_nancy/2046347762)
Why is this different?
Path to data nirvana?
Parallel computing is hard!
Message Passing
P1 P2 P3 P4 P5
Shared Memory
P1 P2 P3 P4 P5
Mem
ory
Different programming models
Different programming constructsmutexes, conditional variables, barriers, …masters/slaves, producers/consumers, work queues, …
Fundamental issuesscheduling, data distribution, synchronization, inter-process communication, robustness, fault tolerance, …
Common problemslivelock, deadlock, data starvation, priority inversion…dining philosophers, sleeping barbers, cigarette smokers, …
Architectural issuesFlynn’s taxonomy (SIMD, MIMD, etc.),network typology, bisection bandwidthUMA vs. NUMA, cache coherence
The reality: programmer shoulders the burden of managing concurrency…(I want my students developing new machine learning algorithms, not debugging race conditions)
Source: Ricardo Guimarães Herrmann
Source: MIT Open Courseware
Source: NY Times (6/14/2006)
The datacenter is the computer!
MapReduce
MapReduce Functional programming meets distributed processing
Independent per-record processing in parallel Aggregation of intermediate results to generate final output
Programmers specify two functions:map (k, v) → <k’, v’>*reduce (k’, v’) → <k’, v’>* All values with the same key are sent to the same reducer
The execution framework handles everything else… Handles scheduling Handles data management, transport, etc. Handles synchronization Handles errors and faults
mapmap map map
Shuffle and Sort: aggregate values by keys
reduce reduce reduce
k1 k2 k3 k4 k5 k6v1 v2 v3 v4 v5 v6
ba 1 2 c c3 6 a c5 2 b c7 8
a 1 5 b 2 7 c 2 3 6 8
r1 s1 r2 s2 r3 s3
split 0split 1split 2split 3split 4
worker
worker
worker
worker
worker
UserProgram
outputfile 0
outputfile 1
(1) submit
(2) schedule map (2) schedule reduce
(3) read(4) local write
(5) remote read(6) write
Inputfiles
Mapphase
Intermediate files(on local disk)
Reducephase
Outputfiles
Adapted from (Dean and Ghemawat, OSDI 2004)
Master
(I want my students developing new machine learning algorithms, not debugging race conditions)
MapReduce Implementations Google has a proprietary implementation in C++
Bindings in Java, Python Hadoop is an open-source implementation in Java
Development led by Yahoo, used in production Now an Apache project Rapidly expanding software ecosystem
Lots of custom research implementations For GPUs, cell processors, etc.
Case Study #1Statistical Machine Translation
Chris Dyer (Linguistics Ph.D., 2010)
Translation Model
LanguageModel
Decoder
Foreign Input Sentence
maria no daba una bofetada a la bruja verde
English Output Sentencemary did not slap the green witch
Word Alignment
Statistical Machine Translation
(vi, i saw)(la mesa pequeña, the small table)…
Phrase Extraction
i saw the small tablevi la mesa pequeña
Parallel Sentences
he sat at the tablethe service was good
Target-Language Text
Training Data
Maria no dio una bofetada a la bruja verde
Mary not
did not
no
did not give
give a slap to the witch green
slap
a slap
to the
to
the
green witch
the witch
by
slap
Translation as a Tiling Problem
Mary
did not
slap
the
green witch
The Data Bottleneck
“Every time I fire a linguist, the performance of our … system goes up.”- Fred Jelinek
Translation Model
LanguageModel
Decoder
Foreign Input Sentence
maria no daba una bofetada a la bruja verde
English Output Sentencemary did not slap the green witch
Word Alignment
Statistical Machine Translation
(vi, i saw)(la mesa pequeña, the small table)…
Phrase Extraction
i saw the small tablevi la mesa pequeña
Parallel Sentences
he sat at the tablethe service was good
Target-Language Text
Training Data
We’ve built MapReduce implementations of these two components!
HMM Alignment: Giza
Single-core commodity server
HMM Alignment: MapReduce
Single-core commodity server
38 processor cluster
HMM Alignment: MapReduce
38 processor cluster
1/38 Single-core commodity server
What’s the point? The optimally-parallelized version doesn’t exist! MapReduce occupies a sweet spot in the design space for
a large class of problems: Fast… in terms of running time + scaling characteristics Easy… in terms of programming effort Cheap… in terms of hardware costs
Chris Dyer, Aaron Cordova, Alex Mont, and Jimmy Lin. Fast, Easy, and Cheap: Construction of Statistical Machine Translation Models with MapReduce. Proceedings of the Third Workshop on Statistical Machine Translation at ACL 2008
Case Study #2DNA Sequence Alignment
Michael Schatz(Computer Science Ph.D., 2010)
Strangely-Formatted Manuscript Dickens: A Tale of Two Cities
Text written on a long spool
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
… With Duplicates Dickens: A Tale of Two Cities
“Backup” on four more copies
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
Shredded Book Reconstruction Dickens accidently shreds the manuscript
How can he reconstruct the text? 5 copies x 138,656 words / 5 words per fragment = 138k
fragments The short fragments from every copy are mixed together Some fragments are identical
It was the best of of times, it was thetimes, it was the worst age of wisdom, it was the age of foolishness, …
It was the best worst of times, it wasof times, it was the the age of wisdom, it was the age of foolishness,
It was the the worst of times, it best of times, it was was the age of wisdom, it was the age of foolishness, …
It was was the worst of times,the best of times, it it was the age of wisdom, it was the age of foolishness, …
It it was the worst ofwas the best of times, times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness, …
It was the best of of times, it was thetimes, it was the worst age of wisdom, it was the age of foolishness, …
It was the best worst of times, it wasof times, it was the the age of wisdom, it was the age of foolishness,
It was the the worst of times, it best of times, it was was the age of wisdom, it was the age of foolishness, …
It was was the worst of times,the best of times, it it was the age of wisdom, it was the age of foolishness, …
It it was the worst ofwas the best of times, times, it was the age of wisdom, it was the age of foolishness, …
Greedy AssemblyIt was the best of
of times, it was the
best of times, it was
times, it was the worst
was the best of times,
the best of times, it
of times, it was the
times, it was the age
It was the best of
of times, it was the
best of times, it was
times, it was the worst
was the best of times,
the best of times, it
it was the worst of
of times, it was the
times, it was the age
it was the age of
was the age of wisdom,
the age of wisdom, it
age of wisdom, it was
of wisdom, it was the
it was the age of
was the age of foolishness,
the worst of times, it
The repeated sequence make the correct reconstruction ambiguous!
Alternative: model sequence reconstruction as a graph problem…
de Bruijn Graph Construction Dk = (V,E)
V = All length-k subfragments (k < l) E = Directed edges between consecutive subfragments
(Nodes overlap by k-1 words)
Locally constructed graph reveals the global structure Overlaps between sequences implicitly computed
It was the best of
Original Fragment
It was the best was the best of
Directed Edge
de Bruijn, 1946Idury and Waterman, 1995Pevzner, Tang, Waterman, 2001
de Bruijn Graph Assembly
the age of foolishness
It was the best
best of times, it
was the best of
the best of times,
of times, it was
times, it was the
it was the worst
was the worst of
worst of times, it
the worst of times,
it was the age
was the age ofthe age of wisdom,
age of wisdom, it
of wisdom, it was
wisdom, it was the
A unique Eulerian tour of the graph reconstructs the
original text
If a unique tour does not exist, try to simplify the
graph as much as possible
de Bruijn Graph Assembly
the age of foolishness
It was the best of times, it
of times, it was the
it was the worst of times, it
it was the age ofthe age of wisdom, it was theA unique Eulerian tour of the
graph reconstructs the original text
If a unique tour does not exist, try to simplify the
graph as much as possible
GATGCTTACTATGCGGGCCCCCGGTCTAATGCTTACTATGC
GCTTACTATGCGGGCCCCTTAATGCTTACTATGCGGGCCCCTT
TAATGCTTACTATGCAATGCTTAGCTATGCGGGC
AATGCTTACTATGCGGGCCCCTTAATGCTTACTATGCGGGCCCCTT
CGGTCTAGATGCTTACTATGCAATGCTTACTATGCGGGCCCCTTCGGTCTAATGCTTAGCTATGC
ATGCTTACTATGCGGGCCCCTT
?Subject genome
Sequencer
Reads
Human genome: 3 gbpA few billion short reads (~100 GB compressed data)
Present solutions: large-shared memory machines or clusters with high-speed interconnects
Can we get by with MapReduce on cheap commodity clusters?
Graph Compression
Challenges– Nodes stored on different machines– Nodes can only access direct neighbors
Randomized Solution– Randomly assign H / T to each
compressible node– Compress H T links
Fast Graph Compression
Initial Graph: 42 nodes
Fast Graph Compression
Round 1: 26 nodes (38% savings)
Fast Graph Compression
Round 2: 15 nodes (64% savings)
Fast Graph Compression
Round 3: 6 nodes (86% savings)
Fast Graph Compression
Round 4: 5 nodes (88% savings)
Contrail De Novo Assembly of the Human Genome
Genome: African male NA18507 (SRA000271, Bentley et al., 2008)
Input: 3.5B 36bp reads, 210bp insert (~40x coverage)
Initial
NMax
>7 B27 bp
Compressed
>1 B303 bp
5.0 M14,007 bp
B
B’
A
Clip Tips
4.2 M20,594 bp
Pop Bubbles
B
B’A C
Assembly of Large Genomes with Cloud Computing.Schatz MC, Sommer D, Kelley D, Pop M, et al. In Preparation.
Source: flickr (fatboyke/2918399820)
Source: flickr (60in3/2338247189)
Best thing since sliced bread? Distributed programming models:
MapReduce is the first Definitely not the only And probably not even the best Alternatives: Pig, Dryad/DryadLINQ, Pregel, etc.
It’s all about the right level of abstraction The von Neumann architecture won’t cut it anymore
Separating the what from how Developer specifies the computation that needs to be performed Execution framework handles actual execution Framework hides system-level details from the developers
Source: NY Times (6/14/2006)
The datacenter is the computer!What are the appropriate abstractions for the datacenter computer?
Source: flickr (infidelic/3008675635)
Source: Wikipedia (Tide)
Commoditization of large-data processing capabilities allows us to ride the rising tide!
Questions?