how to make mpi awesome: a proposal for mpi sessions
TRANSCRIPT
HowtomakeMPIAwesome:MPISessions
WesleyBland,IntelRyanGrant,SandiaNa=onalLaboratory
DanHolmes,EdinburghParallelCompu=ngCenterKathrynMohror,LawrenceLivermoreLaboratoryMar=nSchulz,LawrenceLivermoreLaboratory
AnthonySkjellum,AuburnUniversityJeffSquyres,CiscoSystems,Inc.
^
more
Follow-ontoJeff’scrazythoughtsdiscussedatEuroMPI2015inBordeaux,France
Whatwewant
• FixMPI-3.1limita=ons:– Cannotini=alizeMPIfromdifferenten==eswithinaprocesswithoutaprioriknowledge/coordina=on
– Cannotini=alizeMPImorethanonce– Cannotre-ini=alizeMPIa\erithasbeenfinalized– CannotseterrorbehaviorofMPIini=aliza=on
Whatwewant• Anythread(e.g.,library)canuseMPIany=meitwants• Buts=llbeabletototallycleanupMPIif/whendesired
MPIProcess
// Library 3MPI_Init(…);
// Library 4MPI_Init(…);
// Library 5MPI_Init(…);
// Library 6MPI_Init(…);// Library 7
MPI_Init(…);
// Library 8MPI_Init(…);
// Library 9MPI_Init(…);
// Library 10MPI_Init(…);
// Library 11MPI_Init(…);
// Library 12MPI_Init(…);// Library 1
MPI_Init(…);// Library 2MPI_Init(…);
Howdowegetthosethings?
Newconcept:“session”
• AlocalhandletotheMPIlibrary– Implementa=onintent:lightweight/usesveryfewresources
– Canalsocachesomelocalstate
• Canhavemul=plesessionsinanMPIprocess– MPI_Session_init(…,&session);– MPI_Session_finalize(…,&session);
MPISession
MPIProcess
oceanlibrary
MPI_SESSION_INIT(…)
atmospherelibrary
MPI_SESSION_INIT(…)
MPIlibrary
MPISession
MPIProcess
oceanlibrary
atmospherelibrary
MPIlibrary
oceansession
atmos-pheresession
UniquehandlestotheunderlyingMPIlibrary
MPISession
MPIProcess
oceanlibrary
atmospherelibrary
MPIlibrary
oceanErrorsreturn
atmos-phereErrorsabort
Uniqueerrorhandlers,info,localstate,etc.
Great.Ihaveasession.Nowwhat?
Fairwarning
• TheMPIrun=mehaslong-sincebeenabastardstepchild– Barelyacknowledgedinthestandard
– Mainlyintheformofnon-norma=vesugges=ons
• It’s&metochangethat
Workwiththerun=me
• Generalscheme:– Querytheunderlyingrun-=mesystem• Geta“set”ofprocesses
– Determinetheprocessesyouwant• CreateanMPI_Group
– Createacommunicatorwithjustthoseprocesses• CreateanMPI_Comm
Queryrun=meforsetofprocesses
MPI_Group
MPI_Comm
MPI_Session
Run=meexposessetsofprocesses
• Setsareiden=fiedbystringname• Twosetsaremandated– “mpi://WORLD”– “mpi://SELF”
• Otherimplementa=on-definedsetscanbereported,too
Examplesofsets
MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3
mpi://WORLD
Examplesofsets
MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3
mpi://WORLD
arch://x86_64
Examplesofsets
MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3
mpi://WORLD
job://12942
arch://x86_64
Examplesofsets
MPIprocess0 MPIprocess1 MPIprocess2 MPIprocess3
loca=on://rack/17 loca=on://rack/23
Well,thatallsoundsgreat.
…butwhocallsMPI_INIT?
AndwhatsessiondoesMPI_COMM_WORLD/
MPI_COMM_SELFbelongto?
Newconcept:nolongerrequireMPI_INIT/MPI_FINALIZE
Theoveralltheme
• JustuseMPIfunc=onswheneveryouwant– Ini=aliza=onessen=allybecomesanimplementa=ondetail
• Finaliza=onwilloccurwheneveralluser-definedhandlesaredestroyed
Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype);
Thecrea=onofthefirstuser-definedMPIobjectini=alizesMPI
Ini=aliza=oncanbealocalac=on!
Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype); // Free the datatype – finalizes MPI MPI_Type_free(&mytype);}
Thedestruc=onofthelastuser-definedMPIobjectfinalizesMPI.
Exampleint main() { // Create a datatype – initializes MPI MPI_Type_contiguous(2, MPI_INT, &mytype); // Free the datatype – finalizes MPI MPI_Type_free(&mytype);
// Re-initialize MPI! MPI_Type_dup(MPI_INT, &mytype);
Wecanalsore-ini=alizeMPI!(it’stransparenttotheuser–sowhynot?)
Keyinsight:SplitMPIAPIsintotwosets
Performancedoesn’tma/er(asmuch)
• Func=onsthatcreate/query/destroy:– MPI_Comm– MPI_File– MPI_Win– MPI_Info– MPI_Op– MPI_Errhandler– MPI_Datatype– MPI_Group– MPI_Session– Arributes– Processes
• MPI_T
Performanceabsolutelyma/ers
• Pointtopoint• Collec=ves• I/O• RMA• Test/Wait• Handlelanguagexfer
Keyinsight:SplitMPIAPIsintotwosets
Performancedoesn’tma/er(asmuch)
• Func=onsthatcreate/query/destroy:– MPI_Comm– MPI_File– MPI_Win– MPI_Info– MPI_Op– MPI_Errhandler– MPI_Datatype– MPI_Group– MPI_Session– Arributes– Processes
• MPI_T
Performanceabsolutelyma/ers
• Pointtopoint• Collec=ves• I/O• RMA• Test/Wait• Handlelanguagexfer
EnsurethatMPIisini=alized(and/orfinalized)bythese
func=ons
Thesefunc=onss=llcan’tbeusedunlessMPIis
ini=alized
Keyinsight:SplitMPIAPIsintotwosets
Performancedoesn’tma/er(asmuch)
• Func=onsthatcreate/query/destroy:– MPI_Comm– MPI_File– MPI_Win– MPI_Info– MPI_Op– MPI_Errhandler– MPI_Datatype– MPI_Group– MPI_Session– Arributes– Processes
• MPI_T
Performanceabsolutelyma/ers
• Pointtopoint• Collec=ves• I/O• RMA• Test/Wait• Handlelanguagexfer
Thesefunc=onsinit/finalizeMPItransparently
Thesefunc=onscan’tbecalledwithoutahandlecreatedfrom
thele\-handcolumn
Waitaminute–WhataboutMPI_COMM_WORLD?
int main() { // Can’t I do this? MPI_Send(…, MPI_COMM_WORLD);
Thiswouldbecallinga“performancemarers”
func=onbeforea“performancedoesn’tmarer”
func=on
I.e.,MPIhasnotini=alizedyet
Waitaminute–WhataboutMPI_COMM_WORLD?
int main() { // This is valid MPI_Init(NULL, NULL); MPI_Send(…, MPI_COMM_WORLD);
Re-defineMPI_INITandMPI_FINALIZE:constructoranddestructorfor
MPI_COMM_WORLDandMPI_COMM_SELF
INITandFINALIZE
• INIT/FINALIZEcreateanimplicitsession– YoucannotextractanMPI_SessionhandlefortheimplicitsessioncreatedbyMPI_INIT[_THREAD]
• Yes,youcanuseINIT/FINALIZEinthesameMPIprocessasothersessions
(Abbreviated)Summary
• MPIsession:aunitofisola=on– Allow“ocean”and“atmosphere”scenarios
• Definesomeinterac=onswiththerun=me– Queryprocesssets,makegroups,makecommunicators
• NolongerrequireMPI_INIT/FINALIZE– Keyinsight:cannotinvoke“performancemarers”func=onswithoutahandle
Moreideasanddetailsinthefullslidedeck
hrp://blogs.cisco.com/performance/“MPISessions:
AproposaltotheMPIForum”BlogentryfromMarch2,2016