basic architecture and tuning - open mpi · may 2008 screencast: basic architecture and tuning 17...
TRANSCRIPT
![Page 1: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/1.jpg)
May 2008 Screencast: Basic Architecture and Tuning 1
Screencast: Basic Architecture and Tuning
Jeff Squyres May 2008
![Page 2: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/2.jpg)
May 2008 Screencast: Basic Architecture and Tuning 2
Open MPI Architecture
• Modular component architecture (MCA) Backbone plugin / component system Finds, loads, parameterizes components
• Hierarchy MCA: foundation Framework: functionality specification Component: code for specific functionality Module: “instance” of a component
![Page 3: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/3.jpg)
May 2008 Screencast: Basic Architecture and Tuning 3
User application
MPI API
Modular Component Architecture (MCA)
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
Framework
Com
p.
Com
p.
Com
p.
…
…
Open MPI Architecture
![Page 4: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/4.jpg)
May 2008 Screencast: Basic Architecture and Tuning 4
Three Main Code Sections
• Open MPI layer (OMPI) Top-level MPI API and supporting logic
• Open Run-Time Environment (ORTE) Interface to back-end run-time system
• Open Portability Access Layer (OPAL) OS / utility code (lists, reference counting, etc.)
• Dependencies - not layers OMPI ➔ ORTE ➔ OPAL
![Page 5: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/5.jpg)
May 2008 Screencast: Basic Architecture and Tuning 5
Three Main Code Sections
Operating system
OMPI
ORTE
OPAL
![Page 6: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/6.jpg)
May 2008 Screencast: Basic Architecture and Tuning 6
Three Main Code Sections
Operating system
OMPI ORTE
OPAL
User application
MPI API
![Page 7: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/7.jpg)
May 2008 Screencast: Basic Architecture and Tuning 7
Tuning
![Page 8: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/8.jpg)
May 2008 Screencast: Basic Architecture and Tuning 8
MCA Parameters
• Run-time tunable values Per layer Per framework Per component
• Change behaviors of code at run-time Does not require recompiling / re-linking
• Simple example Choose which network to use for MPI
communications
![Page 9: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/9.jpg)
May 2008 Screencast: Basic Architecture and Tuning 9
MCA Parameter Lookup Order
1. mpirun command line
2. Environment variable
3. File $HOME/.openmpi/mca-params.conf $prefix/etc/openmpi-mca-params.conf
(these locations are themselves tunable) 4. Default value
mpirun --mca <name> <value>
export OMPI_MCA_<name>=<value>
![Page 10: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/10.jpg)
May 2008 Screencast: Basic Architecture and Tuning 10
So Much Information…
• Open MPI has: ~30 frameworks 100+ components Each component has run-time tunable
parameters
• How to know what to use / how to use it?
![Page 11: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/11.jpg)
May 2008 Screencast: Basic Architecture and Tuning 11
ompi_info Command
• Tells everything about OMPI installation Finds all components and all params Great for debugging
• Can look up specific component
Shows parameters and current values Can also use keyword “all”
• “--parsable” option
ompi_info --param <type> <plugin>
![Page 12: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/12.jpg)
May 2008 Screencast: Basic Architecture and Tuning 12
Example: Specify BTL
• BTL: Byte Transfer Layer Framework for MPI point-to-point
communications Select which network to use for MPI
communications
• Framework-level MCA parameter Specifies which components to load
mpirun --mca btl tcp,self \ -np 4 ring_c
![Page 13: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/13.jpg)
May 2008 Screencast: Basic Architecture and Tuning 13
Example: Specify TCP BTL
• Components tcp: TCP sockets self: Process loopback (send-to-self)
mpirun --mca btl tcp,self -np 4 ring_c
![Page 14: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/14.jpg)
May 2008 Screencast: Basic Architecture and Tuning 14
Example: Specify openib BTL
• Components openib: OpenFabrics verbs self: Process loopback (send-to-self)
mpirun --mca btl openib,self -np 4 ring_c
![Page 15: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/15.jpg)
May 2008 Screencast: Basic Architecture and Tuning 15
Example: Specify sm+openib BTLs
• Components openib: OpenFabrics verbs self: Process loopback (send-to-self) sm: Shared memory (on-host communication)
mpirun --mca btl sm,openib,self -np 4 ring_c
![Page 16: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/16.jpg)
May 2008 Screencast: Basic Architecture and Tuning 16
What Does This Do?
mpirun -np 4 ring_c
![Page 17: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/17.jpg)
May 2008 Screencast: Basic Architecture and Tuning 17
What Does This Do?
• Use all available components tcp, sm, openib, …
• TCP too? Yes -- and no TCP will automatically disable itself in the
presence of low latency components (e.g., openib)
mpirun -np 4 ring_c
![Page 18: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/18.jpg)
May 2008 Screencast: Basic Architecture and Tuning 18
What Does This Do?
• More specifically: Open each BTL component Query if it wants to be used Keep all that say “yes” Rank by bandwidth and latency rating
mpirun -np 4 ring_c
![Page 19: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/19.jpg)
May 2008 Screencast: Basic Architecture and Tuning 19
What Does This Do?
mpirun -np 4 --mca btl ^tcp ring_c
![Page 20: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/20.jpg)
May 2008 Screencast: Basic Architecture and Tuning 20
What Does This Do?
• Use all available components except tcp • More specifically:
Open each BTL component except tcp Query if it wants to be used Keep all that say “yes” Rank by bandwidth and latency rating
mpirun -np 4 --mca btl ^tcp ring_c
![Page 21: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/21.jpg)
May 2008 Screencast: Basic Architecture and Tuning 21
openib BTL Parameters
• Shows all openib BTL MCA parameters …there are a lot!
• Also try:
• What do they all mean?
ompi_info --param btl openib
ompi_info --param btl openib \ --parsable
![Page 22: Basic Architecture and Tuning - Open MPI · May 2008 Screencast: Basic Architecture and Tuning 17 What Does This Do? • Use all available components tcp, sm, openib, … • TCP](https://reader035.vdocuments.mx/reader035/viewer/2022071407/60fd9cff45113b2bd4341abb/html5/thumbnails/22.jpg)
May 2008 Screencast: Basic Architecture and Tuning 22