Download - Patterns of parallel programming
Agenda
• Why parallel?• Terms and measures• Building Blocks• Patterns overview
– Pipeline and data flow– Producer-Consumer– Map-Reduce– Other
Why Moore's law is not working anymore
• Power consumption• Wire delays• DRAM access latency• Diminishing returns of more instruction-level
parallelism
Power consumption
10,000
1,000
100
10
1
‘70 ‘80 ’90 ’00 ‘10
Pow
er D
ensi
ty (
W/c
m2 )
8080
Pentium® processors
Hot Plate
Nuclear Reactor
Rocket Nozzle
Sun’s Surface
Wire delays
Diminishing returns
• 80’s– 10 CPI 1 CPI
• 90– 1 CPI 0.5CPI
• 00’s: multicore
No matter how fast processors get, software consistently finds new ways to eat up the extra speed.
Herb Sutter
To scale performance, put many processing cores on the microprocessor chip
New Moore’s law edition is about doubling of cores.
Survival
Terms & Measures• Work = T1• Span = T∞• Work Law: Tp>=T1/P• Span Law: Tp>=T∞• Speedup: Tp/T1
– Linear: θ(P)– Perfect: P
• Parallelism: T1/T∞• Tp<=(T1-T∞)/P + T∞
Definitions
• Concurrent- Several things happenings at the same time
• Multithreaded– Multiple execution contexts
• Parallel– Multiple simultaneous computations
• Asynchronous– Not having to wait
Dangers
• Race Conditions• Starvations• Deadlocks• Livelock• Optimizing compilers • …
Data parallelism
Parallel.ForEach(letters, ch => Capitalize(ch));
Task parallelism
Parallel.Invoke(() => Average(), () => Minimum() …);
Fork-Join• Additional work may be started only when specific subsets of the
original elements have completed processing• All elements should be given the chance to run even if one
invocation fails (Ping)
Fork
Compute Mean
Compute Median
Join
Compute Mode
Parallel.Invoke(() => ComputeMean(),() => ComputeMedian(),() => ComputeMode());
static void MyParallelInvoke(params Action[] actions){
var tasks = new Task[actions.Length];for (int i = 0; i < actions.Length; i++)
tasks[i] = Task.Factory.StartNew(actions[i]);Task.WaitAll(tasks);
}
Pipeline pattern
Task 1
Task 2
Task 3
Task<int> T1 = Task.Factory.StartNew(() => { return result1(); });
Task<double> T2 = T1.ContinueWith((antecedent) => { return result2(antecedent.Result); });
Task<double> T3 = T2.ContinueWith((antecedent) => { return result3(antecedent.Result); });
Producer/Consumer
BlockingCollection<T>
Read 1 Read 2 Read 3Disk/Net
Process Process Process
Other patterns
• Speculative Execution• APM (IAsyncResult, Begin/end pairs)• EAP(Operation/Callback pairs)
References• Patterns for Parallel Programming: Understanding and Applying Par
allel Patterns with the .NET Framework 4• Pluralsight:
– Introduction to Async and Parallel Programming in .NET 4 – Async and Parallel Programming: Application Design
• The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software
• Chapter 27 Multithreaded Algorithms from Introduction to algorithms 3rd edition
