overview of mpi overview of isp’s handling of mpi
DESCRIPTION
Overview of MPI Overview of ISP’s Handling of MPI. About 30 minutes – by Mike. The Continued and Foreseen Ubiquity of MPI . IBM Blue Gene (Picture Courtesy IBM). LANL’s Petascale machine “Roadrunner ” (AMD Opteron CPUs and IBM PowerX Cell). - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/1.jpg)
Overview of MPI
Overview of ISP’s Handling of MPI
About 30 minutes – by Mike
1
![Page 2: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/2.jpg)
2
IBM Blue Gene(Picture Courtesy IBM)
LANL’s Petascale machine“Roadrunner”(AMD Opteron CPUs and IBM PowerX Cell)
• The choice for large-scale parallel simulations (earthquake, weather..)
• Runs “everywhere” – esp on expensive state-of-the-art supercomputers
• Very mature codes exist in MPI – cannot easily be re-implemented• Performs critical simulations in science and engineering• ISP supports dynamic verification of MPI C applications
The Continued and Foreseen Ubiquity of MPI
![Page 3: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/3.jpg)
Overview of Message Passing Interface (MPI) API• One of the Stunning Standardization Successes
• Lingua franca of Parallel Computing
• Runs on parallel machines of a WIDE range of sizes
• Standard is published at www.mpi-forum.org
• MPI 2.0 includes over 300 functions
![Page 4: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/4.jpg)
Overview of ISP’s Handling of MPI Calls
• MPI 2.0 includes over 300 functions
• ISP works by Hijacking the MPI Calls
• ISP currently handles over 60 Calls– Includes very popular ones– ISP has FULLY handled MANY applications (next slide)– We are continually extending the range– Calls not handled can often be directly issued (no hijack)
• Creating a tool such as ISP is a non-trivial venture– Ongoing research extends ISP’s range
![Page 5: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/5.jpg)
ISP’s Score-Card of handling MPI / C Applications
![Page 6: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/6.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
![Page 7: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/7.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
![Page 8: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/8.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine
![Page 9: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/9.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
![Page 10: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/10.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner
![Page 11: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/11.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
![Page 12: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/12.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST
![Page 13: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/13.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST• Runs to completion on small instances
![Page 14: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/14.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST• Runs to completion on small instances
• A few MPI Spec Benchmarks
![Page 15: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/15.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST• Runs to completion on small instances
• A few MPI Spec Benchmarks• Some benchmarks exhibit interleaving explosion; others
OK
![Page 16: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/16.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST• Runs to completion on small instances
• A few MPI Spec Benchmarks• Some benchmarks exhibit interleaving explosion; others
OK• ADLB
![Page 17: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/17.jpg)
ISP’s Score-Card of handling MPI / C Applications• Game of Life
• EuroPVM / MPI 2007 versions of Gropp and Lusk done in seconds
• MADRE – Siegels’ Mem Aware Data Redistrib. Engine• Found previously documented deadlock
• ParMETIS – Hypergraph Partitioner• Initial run of days reduced now to seconds on a laptop
• MPI-BLAST – Genome sequencer using BLAST• Runs to completion on small instances
• A few MPI Spec Benchmarks• Some benchmarks exhibit interleaving explosion; others
OK• ADLB
• Initial experiments have been successful
![Page 18: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/18.jpg)
18
Crash course on MPI
If you don’t know MPI, you’ll like our brevity
If you know MPI, then too you’ll like the brevity
![Page 19: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/19.jpg)
19
We will introduce four MPI functions
Namely S, R, W, and B
(out of the > 300)
![Page 20: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/20.jpg)
Non-blocking send, MPI_Isend• MPI_Isend(destination, msg_buf, req_struct)• This is a non-blocking call
• Start copying out msg_buf• Starts a memory-to-memory copy from issuing
process to destination process– May benefit from runtime buffering
• MPI_Wait(req_struct) awaits completion– When Wait unblocks, sending proc can reuse
msg_buf
20
![Page 21: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/21.jpg)
21
We often abbreviate MPI_Isendas Isend or simply S
We often abbreviate MPI_Waitas Wait or simply W
![Page 22: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/22.jpg)
Non-blocking receive, MPI_Irecv• MPI_Irecv(source, msg_bug, req_struct, ..)• This is a non-blocking receive call
• Starts a memory-to-memory copy from source process• Source can be specified as ANY-SRC or ‘wildcard’ or ‘*’
• MPI_Wait(req_struct) awaits completion– When Wait unblocks, msg_buff is ready for consumption
• Source can be – “wildcard” or * or ANY_SOURCE– Receive from any eligible (matching) sender
22
![Page 23: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/23.jpg)
23
We abbreviate MPI_Irecv as Irecv, or simply R
![Page 24: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/24.jpg)
24
Illustration of S and R in action
in ‘lucky.c’ and ‘unlucky.c’
![Page 25: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/25.jpg)
25
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
//Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
Example MPI program ‘lucky.c’
![Page 26: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/26.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
//Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
26
Example MPI program ‘lucky.c’
![Page 27: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/27.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
//Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
27
deadlock
Example MPI program ‘lucky.c’ (lucky for tester)
![Page 28: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/28.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
// Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
28
MPI program ‘unlucky.c’
![Page 29: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/29.jpg)
Process P0
R(from:*, r1) ;
R(from:2, r2);
S(to:2, r3);
R(from:*, r4);
All the Ws…
Process P1
// Sleep(3);
S(to:0, r1);
All the Ws…
Process P2
Sleep(3);
S(to:0, r1);
R(from:0, r2);
S(to:0, r3);
All the Ws…
29
No deadlock
‘unlucky.c’
![Page 30: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/30.jpg)
More MPI Commands
– MPI_Barrier(…) is abbreviated as Barrier() or B
– All processes must invoke Barrier before any process can return from Barrier
call
– Useful high-performance global sync. operation
– Sometimes used ‘for fear of the unforeseen’• ISP’s algorithm for removing the Functionally
Irrelevant Bs, or F.I.B. (see EuroPVM/MPI 2008)
30
![Page 31: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/31.jpg)
31
So you think you really understand
S, R, W, and B ?
Let us find out!
![Page 32: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/32.jpg)
MPI Quiz involving S, R, W, and B
Will this single-process example called “Auto-send” deadlock ?
P0 : R(from:0, h1); B; S(to:0, h2); W(h1); W(h2);
![Page 33: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/33.jpg)
Types of bugs detected by ISP• Deadlocks
– Quite common• Due to mismatched sends / receives
– Unequal source / destination field– Unequal tags– Different communicators
• Mismatched collectives– Flags many that are ‘glibly handled’ by real MPI runtimes
» Example : MPI_Allreduce : root must participate too• Or else, we must deadlock – yet MPI libraries often don’t
– Yet easily missed in the exploding number of schedules!
– ISP GUI allows users to identify onset of deadlock in replay trace
– ISP finds deadlocks by• Generating only the relevant interleavings• Ensuring that ALL the above sources will be detected!
33
![Page 34: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/34.jpg)
Types of bugs detected by ISP• MPI Object Leaks
– Quite common• Due to forgotten deallocations of one or more of these :
– Communicators– Request Structures– Types– …
• Causes ‘slow death’ of important programs!– Nightmarish to debug !
– Yet easily missed in the exploding number of schedules!
– ISP GUI takes you to the allocation that was leaked
– ISP finds leaks• Generating only the relevant interleavings• Instrumenting uses
34
![Page 35: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/35.jpg)
Types of bugs detected by ISP• Assertion Violations– Programmers MUST use C ‘assert’ statements
– ISP GUI takes you to failing assert
– Guaranteed checking• Generating only the relevant interleavings• Checks across ALL these – won’t miss assert
checking due to “chancy” speed-dependent scheduling
35
![Page 36: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/36.jpg)
Types of bugs detected by ISP• Default Safety Properties–Many MPI usage-checking rules are yet to
be built–Will make ISP all the more powerful
36
![Page 37: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/37.jpg)
Summary of ISP• We have built the only push-button dynamic analysis tool for MPI / C
programs called ISP• Work on MPI / Fortran in progress• Runs on MAC OS/X, Windows, Linux• Tested against five state-of-the-art MPI libraries
• MPICH2, OpenMPI, MSMPI, MVAPICH, IBM MPI (in progress)• Visual-Studio and Eclipse Parallel Tools Platform integration• 100s of large case studies : tool + these + LiveDVD ISO
available !!• Guarantees to find assertion violations, deadlocks, MPI leaks (for a
given test harness, all RELEVANT interleavings are replayed)• Efficiency is decent (getting better)
• 15K LOC Parmetis Hypergraph Partitioner analyzed for deadlocks, resource leaks, assertion violations for a given test harness in < 5 seconds for 2 MPI processes on a laptop
• Contribution to the Eclipse Consortium underway (lawyers looking now)
• ISP can dynamically execute and reveal the space of all standard-compliant executions of MPI even when running on an arbitrary platform• ISP’s internal scheduling decisions are taken in a fairly general way
![Page 38: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/38.jpg)
38
(BlueGene/L - Image courtesy of IBM / LLNL)
(Image courtesy of Steve Parker, U of Utah)
• Verifies MPI User Applications, generating
only the Relevant Process Interleavings • Detects all Deadlocks, Assert Violations, MPI object leaks, and Default Safety Properties• Works by Instrumenting MPI Calls Computing Relevant Interleavings, Replaying
Features of ISP, dynamic verifier for MPI apps
![Page 39: Overview of MPI Overview of ISP’s Handling of MPI](https://reader035.vdocuments.mx/reader035/viewer/2022081421/568164aa550346895dd6a728/html5/thumbnails/39.jpg)
End of B
39