parallel programming abstractions
DESCRIPTION
Parallel Programming Abstractions. Tasks vs Threads. Similar but not the same. T asks. Task Scheduler. T hreads. Operating System. h/w processors. Tasks vs Threads: Main Differences. Tasks need not run concurrently The task scheduler can schedule them on the same thread. - PowerPoint PPT PresentationTRANSCRIPT
Parallel Programming Abstractions 1
PARALLEL PROGRAMMING ABSTRACTIONS
6/16/2010
Parallel Programming Abstractions 2
Tasks vs Threads• Similar but not the same.
6/16/2010
h/w processors
Operating System
Threads
Task Scheduler
Tasks
Parallel Programming Abstractions 3
Tasks vs Threads: Main Differences• Tasks need not run concurrently• The task scheduler can schedule them on the same thread
6/16/2010
Parallel Programming Abstractions 4
Tasks vs Threads: Main Differences• Tasks need not run concurrently• The task scheduler can schedule them on the same thread
• Tasks do not have fairness guarantees
6/16/2010
while(t == 0);Write(“hello”);
t = 1;
Parallel Programming Abstractions 5
Tasks vs Threads: Main Differences• Tasks need not run concurrently• The task scheduler can schedule them on the same thread
• Tasks do not have fairness guarantees
• Tasks are cheaper than threads• Have smaller stacks• Do not pre-allocate OS resources
6/16/2010
Parallel Programming Abstractions 6
Generating Tasks from Programs• You don’t want programmers to explicitly create tasks• Like assembly level programming
• Instead:• Design programming language constructs that capture programmers
intent• Compiler converts these constructs to tasks
• Languages with the following features are very convenient for this purpose• Type inference• Generics• First order anonymous functions (lamdas/delegates)
6/16/2010
Parallel Programming Abstractions 7
First-order functions: C# Lambda Expressions • Syntax (input parameters) => expression
• Is similar to : &anon_foo // where foo is declared elsewhere anon_foo(input parameters){ return expression;}
• Examples:x => x(x,y) => x==y(int x, string s) => s.Length > x() => { return SomeMethod()+1; }
Func<int, bool> myFunc = x => x == 5;bool result = myFunc(4);
6/16/2010
Parallel Programming Abstractions 8
Sequential Merge Sort With Lambdas
6/16/2010
MergeSort(int[] a, low, hi){ if(base_case) …
int mid = low + (hi-low)/2;
var f = (l,h) => { MergeSort(a, l, h);}
f(low, mid-1); f(mid, high); Merge(a, low, mid, hi);}
Parallel Programming Abstractions 9
Things to Know about C# Lambdas• Lambda is an expression (with no type)• Conversion to a delegate type• Type inference for parameters• Capture of free variables• Locations referenced by free variables are converted to be on
the heap (“boxed”)
6/16/2010
Parallel Programming Abstractions 10
The Task Abstraction
6/16/2010
delegate void Action();
class Task {Task( Action a );void Wait();
// called by the WSQ schedulervoid Execute();
}
Parallel Programming Abstractions 11
Merge Sort With Tasks
6/16/2010
MergeSort(int[] a, low, hi){ if(base_case) …
int mid = low + (hi-low)/2;
Task left = new Task( delegate{ MergeSort(a, low, mid); } );
Task right = new Task( delegate{ MergeSort(a, mid, hi); } );
left.Wait(); right.Wait(); Merge(a, low, mid-1, hi);}
Parallel Programming Abstractions 12
Parallel.Invoke
• Invokes all input actions in parallel • Waits for all of them to finish
6/16/2010
static void Invoke(params Action[] actions);
Parallel Programming Abstractions 13
Merge Sort With Parallel.Invoke
6/16/2010
MergeSort(int[] a, low, hi){ if(base_case) …
int mid = low + (hi-low)/2;
Paralle.Invoke{ () => { MergeSort(a, low, mid-1); } () => { MergeSort(a, mid, hi); } } Merge(a, low, mid, hi);}
Parallel Programming Abstractions 14
Compare with Sequential Version
6/16/2010
MergeSort(int[] a, low, hi){ if(base_case) …
int mid = low + (hi-low)/2;
{ MergeSort(a, low, mid-1); } { MergeSort(a, mid, hi); } Merge(a, low, mid, hi);}
Parallel Programming Abstractions 15
Data Parallelism• Sometimes you want to perform the same
computation on all elements of a collection
• For every string in an array, check if it contains “foo”
6/16/2010
Parallel Programming Abstractions 16
Parallel.For
• Iterates a variable i from lower from to upper• Calls the delegate with i as the parameter in parallel
6/16/2010
For(int lower, int upper, delegate int (int) body);
Parallel Programming Abstractions 17
Parallel.For
6/16/2010
// sequential forfor(int i=0; i<n; i++){ if(a[i].Contains(“foo”)){DoSomething(a[i]);}}
//Parallel forParallel.For(0, n, (i) => { if(a[i].Contains(“foo”)){DoSomething(a[i]);}});
Parallel Programming Abstractions 18
A;Parallel.For(0, N, m: i => { B; });C;
m(0) m(1) m(N-1)…
The DAG created by Parallel.For
A
C
6/16/2010
Parallel Programming Abstractions 19
Paralle.ForEach
• Same as Parallel.For, but iterates over elements of a collection in parallel
6/16/2010
ForEach<T>(IEnumerable<T> source, delegate void (T) body);
Parallel Programming Abstractions 20
Advantage of High-Level Abstractions• Makes it easy to add parallelism• Explore and experiment with different parallelization
strategies
• Language Compiler/Runtime can perform performance optimizations• Efficiently create tasks• Efficiently distribute tasks to available cores• Tight integration with scheduler
6/16/2010
Parallel Performance 21
Example Partitioning on Two Cores6/16/2010
Parallel Performance 22
Partitioning on Four Cores6/16/2010
Parallel Programming Abstractions 23
Advantage of High-Level Abstractions• Makes it easy to add parallelism• Explore and experiment with different parallelization strategies
• Language Compiler/Runtime can perform performance optimizations• Efficiently create tasks• Efficiently distribute tasks to available cores• Tight integration with scheduler
• Provide programmatic features• Exceptions• Cancelling running tasks
6/16/2010
Parallel Programming Abstractions 24
Semantics of Parallel Constructs• What does this do:
6/16/2010
Paralle.Invoke{ () => { WriteLine(“Hello”); } () => { WriteLine(“World”); } }
Parallel Programming Abstractions 25
Semantics of Parallel Constructs• What does this do:
• Compare with
6/16/2010
Paralle.Invoke{ () => { WriteLine(“Hello”); } () => { WriteLine(“World”); } }
{ WriteLine(“Hello”); } { WriteLine(“World”); }
Parallel Programming Abstractions 26
Semantics of Parallel Constructs• By writing this program
• You are telling the compiler that both outcomes are acceptable
6/16/2010
Paralle.Invoke{ () => { WriteLine(“Hello”); } () => { WriteLine(“World”); } }
Parallel Programming Abstractions 27
Correctness Criteria• Given a DAG, any linearization of the DAG is an
acceptable execution
6/16/2010
Parallel Programming Abstractions 28
Correctness Criteria• Simple criterion:• Every task operates on separate data• No dependencies other than the edges in the DAG
• We will look at more complex criteria later in the course.
6/16/2010