programming clusters with dryadlinq mihai budiu microsoft research, silicon valley association of c...
Post on 16-Jan-2016
226 Views
Preview:
TRANSCRIPT
Programming clusters with DryadLINQ
Mihai BudiuMicrosoft Research, Silicon Valley
Association of C and C++ Users (ACCU)Mountain View, CA, April 13, 2011
2
Goal
3
Design Space
Throughput(batch)
Latency(interactive)
Internet
Datacenter
Data-parallel
Sharedmemory
DryadSearch
HPC
Grid
Transaction
Execution
Application
Data-Parallel Computation
4
Storage
Language
ParallelDatabases
Map-Reduce
GFSBigTable
CosmosAzure
SQL Server
Dryad
DryadLINQScope
Sawzall,FlumeJava
Hadoop
HDFSS3
Pig, HiveSQL ≈SQL LINQ, SQLSawzall, Java
5
Software Stack: Talk Outline
Windows Server
Cluster services
Cluster storage
Dryad
DryadLINQ
Windows Server
Windows Server
Windows Server
Applications
6
DRYAD
Windows Server
Cluster services
Cluster storage
Dryad
DryadLINQ
Windows Server
Windows Server
Windows Server
Applications
7
Dryad
• Continuously deployed since 2006• Running on >> 104 machines• Sifting through > 10Pb data daily• Runs on clusters > 3000 machines• Handles jobs with > 105 processes each• Platform for rich software ecosystem• Used by >> 100 developers
• Written at Microsoft Research, Silicon Valley
The Dryad by Evelyn De Morgan.
8
Dryad = Execution Layer
Job (application)
Dryad
Cluster
Pipeline
Shell
Machine≈
9
2-D Piping• Unix Pipes: 1-D
grep | sed | sort | awk | perl
• Dryad: 2-D grep1000 | sed500 | sort1000 | awk500 | perl50
10
Virtualized 2-D Pipelines
11
Virtualized 2-D Pipelines
12
Virtualized 2-D Pipelines
13
Virtualized 2-D Pipelines
14
Virtualized 2-D Pipelines• 2D DAG• multi-machine• virtualized
15
Dryad Job Structure
grep
sed
sortawk
perlgrep
grepsed
sort
sort
awk
Inputfiles
Vertices (processes)
Outputfiles
ChannelsStage
16
Channels
X
M
Items
Finite streams of items
• distributed filesystem files (persistent)• SMB/NTFS files (temporary)• TCP pipes (inter-machine)• memory FIFOs (intra-machine)
17
Dryad System Architecture
Files, TCP, FIFO, Networkjob schedule
data plane
control plane
NS,Sched RE RERE
V V V
Job manager cluster
Fault Tolerance
19
DRYADLINQ
Windows Server
Cluster services
Cluster storage
Dryad
DryadLINQ
Windows Server
Windows Server
Windows Server
Applications
20
LINQ
Dryad
=> DryadLINQ
21
LINQ = .Net+ Queries
Collection<T> collection;bool IsLegal(Key);string Hash(Key);
var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};
22
Collections and Iterators
class Collection<T> : IEnumerable<T>;
Elements of type TIterator(current element)
23
DryadLINQ Data Model
Partition
Collection
.Net objects
24
Collection<T> collection;bool IsLegal(Key k);string Hash(Key);
var results = from c in collection where IsLegal(c.key) select new { Hash(c.key), c.value};
DryadLINQ = LINQ + Dryad
C#
collection
results
C# C# C#
Vertexcode
Queryplan(Dryad job)Data
25
Demo
Example: counting lines
var table = PartitionedTable.Get<LineRecord>(file);int count = table.Count();
Parse, Count
Sum
Example: counting words
var table = PartitionedTable.Get<LineRecord>(file);int count = table
.SelectMany(l => l.line.Split(‘ ‘))
.Count();
Parse, SelectMany, Count
Sum
Example: counting unique wordsvar table = PartitionedTable.Get<LineRecord>(file);int count = table
.SelectMany(l => l.line.Split(‘ ‘))
.GroupBy(w => w)
.Count();
GroupBy;Count
HashPartition
Example: word histogramvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' ')) .GroupBy(w => w) .Select(g => new { word = g.Key, count = g.Count() });
GroupBy;Count
GroupByCountHashPartition
Example: high-frequency wordsvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' '))
.GroupBy(w => w)
.Select(g => new { word = g.Key, count = g.Count() })
.OrderByDescending(t => t.count)
.Take(100);
Sort; Take Mergesort;
Take
Example: words by frequencyvar table = PartitionedTable.Get<LineRecord>(file);var result = table.SelectMany(l => l.line.Split(' '))
.GroupBy(w => w)
.Select(g => new { word = g.Key, count = g.Count() })
.OrderByDescending(t => t.count);
Sample
Histogram
Broadcast
Range-partition
Sort
Example: Map-Reducepublic static IQueryable<S> MapReduce<T,M,K,S>( IQueryable<T> input,
Func<T, IEnumerable<M>> mapper,Func<M,K> keySelector,Func<IGrouping<K,M>,S> reducer)
{ var map = input.SelectMany(mapper); var group = map.GroupBy(keySelector); var result = group.Select(reducer); return result;}
33
Map-Reduce Plan
M
R
G
M
Q
G1
R
D
MS
G2
R
X
X
M
Q
G1
R
D
MS
G2
R
X
M
Q
G1
R
D
MS
G2
R
X
M
Q
G1
R
D
M
Q
G1
R
D
MS
G2
R
X
M
Q
G1
R
D
MS
G2
R
X
M
Q
G1
R
D
MS
G2
R
MS
G2
R
map
sort
groupby
reduce
distribute
mergesort
groupby
reduce
mergesort
groupby
reduce
consumer
map
parti
al a
ggre
gatio
nre
ducedynamic
Expectation Maximization
34
• 160 lines • 3 iterations shown
35
Probabilistic Index MapsImages
features
36
Language Summary
WhereSelectGroupByOrderByAggregateJoin
37
What Is It Good For?
38
What is Kinect?
39
Input device
40
The Innards
Source: iFixit
41
Projected IR pattern
Source: www.ros.org
42
Depth computation
Source: http://nuit-blanche.blogspot.com/2010/11/unsing-kinect-for-compressive-sensing.html
43
Kinect video output30 HZ frame rate
57deg field-of-view
8-bit VGA RGB640 x 480
11-bit depth320 x 240
44
Depth map
Source: www.insidekinect.com
45
Vision Problem: What is a human
• Recognize players from depth map• At frame rate• Minimal resource usage
46
XBox 360 Hardware
Source: http://www.pcper.com/article.php?aid=940&type=expert
• Triple Core PowerPC 970, 3.2GHz• Hyperthreaded, 2 threads/core
• 500 MHz ATI graphics card• DirectX 9.5
• 512 MB RAM
• 2005 performance envelope
• Must handle real-time vision AND a modern game
Why is it hard?
48
Generic Extensible Architecture
Expert 1
Expert 2
Expert 3
Arbiter
StatelessRawdataSensor Skeleton
estimatesFinal
estimate
probabilistic
fuses the hypotheses
Stateful
Background segmentationPlayer separation
Body Part Classifier
One Expert: Pipeline Stages
49
Depth mapSensor
Body Part Identification
Skeleton
50
Sample test frames
51
The Classifier
InputDepth map
OutputBody parts
Classifier
Runs on GPU @ 320x240
52
• Start from ground-truth data– depth paired with body parts
• Train classifier to work across– pose– scene position– Height, body shape
Getting the Ground Truth
53
Getting the Ground Truth
Use synthetic data (3D avatar model)• Inject noise
Motion capture
[Xsens][Vicon]
suit / sensors expensive
very accurate high frame rate
large space calibration
55
Learn from Data
Classifier
Training examplesMachine learning
56
Cluster-based training
Classifier
Training examples
Dryad
DryadLINQ
Machine learning
• > Millions of input frames• > 1020 objects manipulated• Sparse, multi-dimensional data• Complex datatypes
(images, video, matrices, etc.)
57
Highly efficient parallellization
time
mac
hine
58
CONCLUSIONS
Conclusions
59
Cluster
LINQ
Dryad
59
=
60
I can finally explain to my sonwhat I do for a living…
top related