cnc: a dependence programming model
TRANSCRIPT
![Page 2: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/2.jpg)
Mo@va@onforthetalk(notmo@va@onforCnC)
• SomeonehearsaboutCnC– Tolearnmoretheycheckintoanimplementa@on– Itwasn’tdesignedforwhattheywant– Theywalkaway
• Ourmarke@ngdepartmentneedstomakesomechangestohowwepresent/write/...aboutCnC
• Here’sapossibledifferentslant• YouknowmostofthisbutI’mlookingfor
– Feedback/discussionontheproblem– Feedback/discussiononthestoryhere– Recommenda@onsforchangingthepresenta@on
• Thisseemsliketherightgroup
![Page 3: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/3.jpg)
Performance
Eigensolver
UsedonDARPA:UHPCprojectatIntelDOE:X-StackprojectatIntelDreamworks:
“HowtotrainyourdragonII”
Intel 2-socket x 4-core Nehalem @ 2.8 GHz + Intel® Math Kernel Libraries 10.2
Mul@threadedIntel®MKL
Baseline
ConcurrentCollec@ons
Intel’s MKL team
Aparna Chandramolishwaren One GaTech first year grad student intern at Intel
![Page 4: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/4.jpg)
Cholesky
Intel 2-socket x 4-core Nehalem @ 2.8 GHz + Intel MKL 10.2
Plasma: Jack Dongarra’s group at Oak Ridge NL
Aparna Chandramolishwaren One GaTech first year grad student intern at Intel
![Page 5: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/5.jpg)
Outline
1. Mo@va@on2. CnC3. Widearrayofexis@ngCnCtuningapproaches
![Page 6: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/6.jpg)
Outline
1. Mo@va@on2. CnC3. Widearrayofexis@ngCnCtuningapproaches
![Page 7: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/7.jpg)
Stylesoflanguages/models• SeriallanguagesandParallellanguages
– Orderings:• Required/tuning/arbitrary• Makeamodifica@on:mustdis@nguishamongthese
– Takes@me– Leadstoerrors
• Toofewdependences:errors• Toomanydependences:looseperformance
• Dependencelanguages– Orderings:
• Requiredonly• Notuningorderings• Noarbitraryorderings
– Makeamodifica@on=>theorderingrequirementsareclear• No@me• Noerrors
• CnCisadependenceprogrammingmodel• Thedependencesincludedataandcontrol
dependences- Bothareexplicit- Theyareatthesamelevel- Theyaredis@nct
![Page 8: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/8.jpg)
Somerelevantcurrentapproaches
• Startwithanexplicitlyserialoranexplicitlyparallelprogram.Modifyitforsomespecifictargetorspecificop@miza@ongoal
Havetoundoortripoverearliertunings
• Startwithsourcecodeandautoma@callyuncovertheparallelism
We’vebeenthere
• StartwiththewhiteboarddrawingIt’showwethinkaboutourappsDependencesareexplicitbut:notexecutable
![Page 9: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/9.jpg)
9
Anactualwhiteboardsketch:LULESHashockhydro-dynamicsapp
![Page 10: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/10.jpg)
ACnCgraphofflowofdataforLULESH
![Page 11: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/11.jpg)
UnlikeanactualwhiteboardsketchCnCisaformalmodel
• Itisprecise– Itisexecutable
• Hasenoughinforma@on– Toproveproper@esabouttheprogram– Toanalyzetheprogram– Toop1mizetheprogram
• Atthegraphlevel– Withoutaccesstocomputa@oncode
![Page 12: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/12.jpg)
Outline
1. Mo@va@on2. CnC– Goalsthatsupportexascaleneeds– CnCdetailsviaanexampleapp
3. Widearrayofexis@ngCnCtuningapproaches
![Page 13: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/13.jpg)
Needsforexascale
So5wareengineering• Separa@onofvariousac@vi@es
(waysofthinking)– Thedetailsofthecomputa@on
withinthecomputa@onsteps– Thedependencesamongsteps– Therun@me– Thetuning
• Leadstoimprovedreuse• Domainexpertdoesn’tneed
toknowabout– Architectureissues– Tuninggoals
Tuning• Givenonlydependences• Hidesirrelevantlowlevel
computa@ondetails• Eachnewtuningstartsfrom
justdependences• Useanystyleoftuning
Justobeythedependences• Computerscien@stdoesn’t
needtoknowabout– Physics,economics,
biochemistry
Tuningexpert(computerscien@st).
Domainexpert(physicist,economist,biochemist,…)
SameordifferentpeopleDifferentac@vity
CommunicateatthelevelofthegraphNotaboutphysicsoraboutparallelperformance
![Page 14: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/14.jpg)
Tuning
• Tuningconsumesbulkofthe@meandenergy• ThereisnoCnC-specificapproachtotuning
– CnCisnotaparallelprogrammingmodel• Onlyonerequirement:
– thetunedappmustobeydomainspecseman@cs• Singledomainspec/Awiderangeoftuninggoals&styles
– Faultycomponents– Dis@ncttuninggoals(@me,memory,energy,…)– Manydifferentstylesofrun@mes(sta@c/dynamic)– Manydifferentstylesoftuning(sta@c/dynamic)
• Domainspecisn’tmodifiedbytunings– Tuningsmayrefertodomainspecbutareisolatedfromit
![Page 15: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/15.jpg)
CnCsimplifiestuningbysepara@onofconcerns
• Separatesthedetailsofthecomputa@onsanddatafromthedependencesamongthem
• Domainspecisisolatedfromtuning– Itexplicitlyrepresentstheorderingrequirements– Noorderingsfortuning&noarbitraryorderings
![Page 16: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/16.jpg)
Outline
1. Mo@va@on2. CnC– Goalsthatsupportexascaleneeds– CnCdetailsviaanexampleapp
3. Widearrayofexis@ngCnCtuningapproaches
![Page 17: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/17.jpg)
Choleskyfactoriza@on
17
Cholesky
Trisolve Update
![Page 18: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/18.jpg)
Choleskyfactoriza@on
18
Cholesky
Trisolve Update
![Page 19: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/19.jpg)
Choleskyfactoriza@on
19
Cholesky
Trisolve Update
![Page 20: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/20.jpg)
Choleskyfactoriza@on
20
Cholesky
Trisolve Update
![Page 21: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/21.jpg)
Cholesky
Trisolve Update
Choleskyfactoriza@on
21
Cholesky
Trisolve Update
![Page 22: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/22.jpg)
Choleskyfactoriza@on
22
Cholesky
Trisolve Update
![Page 23: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/23.jpg)
CholeskyCnCdomainspec1:whiteboarddrawing
(C:iter)
(T::iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row] [U:iter,row,col]
Computa@onstepcollec@on
Computa@onstepcollec@on
Computa@onstepcollec@onDataitemcollec@on
Dataitemcollec@on Dataitemcollec@on
Collec@onsarejustsetsofinstances
![Page 24: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/24.jpg)
CholeskyCnCdomainspec1:whiteboarddrawing
(C:iter)
(T::iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row] [U:iter,row,col]
Bydefault:dataitemsaredynamicsingleassignment
Bydefault:computa@onstepsaredeterminis@c
![Page 25: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/25.jpg)
CholeskyCnCdomainspec2:I/O
(C:iter)
(T::iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row] [U:iter,row,col]
![Page 26: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/26.jpg)
CholeskyCnCdomainspec3:Dis@nguishinstances
(C:iter)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row] [U:iter,row,col]
Tagsarejustiden@fiersRequireanequalityoperatorOrenintegers:itera@on,row,col
![Page 27: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/27.jpg)
CholeskyCnCdomainspec3:Dis@nguishinstances
(C:iter)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row] [U:iter,row,col]
Tagsarejustiden@fiersRequireanequalityoperatorOrenintegers:itera@on,row,colButnotnecessarily:
representa@onofaSudokuboard
![Page 28: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/28.jpg)
CholeskyCnCdomainspec4:exactsetofinstances(control)
(C:iter)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row]
<I:iter>
<IR:iter,row>
<IRC:iter,row,col>
[U:iter,row,col]
Controltagcollec@on
Controltagcollec@on
Controltagcollec@on
![Page 29: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/29.jpg)
CholeskyCnCdomainspec4:exactsetofinstances(control)
(C:iter)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row]
<IR:iter,row>
<IRC:iter,row,col>
[U:iter,row,col]
Computa@onstepinstance:In/compute/out/doneNocyclicdependencesonasingleinstance
<I:iter>
![Page 30: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/30.jpg)
CholeskyCnCdomainspec4:exactsetofinstances(control)
(C:iter)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row]
<I:iter>
<IR:iter,row>
<IRC:iter,row,col>
[U:iter,row,col]
Controldependence
Datadependence
![Page 31: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/31.jpg)
CholeskyCnCdomainspecOp@onal:dependencesfunc@ons
(Citer)
(T:iter,row)
(U:iter,row,col)
[C:iter]
[T:iter,row]
<I:iter>
<IR:iter,row>
<IRC:iter,row,col>
[U:iter,row,col]
(C:iter)consumes[U:iter-1,iter,iter]
(C:iter)produces[C:iter]
![Page 32: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/32.jpg)
Seman@csofCnC:specifiesapar@alorderofexecu@on
Item
avail
stepcontrolReady
stepready
stepdataReady
stepexecuted
tagavail
![Page 33: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/33.jpg)
Outline
1. Mo@va@on2. CnC3. Widearrayofexis@ngCnCtuningapproaches
![Page 34: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/34.jpg)
Tuning
Tuningstartswiththedomainspec• Justtherequiredorderings
Widerangeoftuningapproaches• Exis@ngsta@ctunings• Exis@ngdynamictunings• Possiblefuturetunings
![Page 35: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/35.jpg)
Sta@ctuning
• Totallysta@cscheduleacross@meandspace• Automa@cconversionofelement-leveldataand
computa@onto@ledversions• Polyhedral@ling• Distribu@onfunc@onsacrossnodesinacluster
(dynamicwithineachnode)– Fordataitemsor– Forcomputa@onsteps
• MinimizeRToverheadbyelimina@ngredundantauributepropaga@on
• PIPESfordistributedapps(Sta@canddynamic)
CarlOffnerAlexNelson
AlinaSbirlea
AlinaSbirleaJunShirako
FrankSchlimbach
ZoranBudimlićKathKnobe
TiagoCogumbreiro
VivekSarkar
Mar@nCongYuhanPeng
![Page 36: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/36.jpg)
Dynamictuning• Run@mesbasedonOCR,TBB,BabelandHaskell
• Basic:determineswhenastepisready– TracksstatechangesandexecutesREADYsteps
• Workflowcoordina@on• ChoiceamongCPU,GPU,FPGA
– Dynamiccompromise:Sta@csteppreferenceanddynamicdeviceavailability
• Hierarchicalaffinitygroupsforlocality• Memoryusage:
Dynamicsingleassignmentishigherlevel:Iden@fiesvalues,notplacesMappingdatavaluestomemoryistuning
– Garbagecollec@on:Get-count– Inspector/executor
RyanNewton
ShamsImam
NickVrviloZoranBudimlićVivekSarkar
FrankSchlimbach
ZoranBudimlićMikeBurke
SanjayChauerjeeKathKnobeNickVrvilo
AlinaSbirleaZoranBudimlić
RiceteamFrankSchlimbach
DragosSbirlea
FrankSchlimbach
FrankSchlimbachYvesVandriessche
VivekSarkar
![Page 37: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/37.jpg)
Possiblefuturetunings• Out-of-corecomputa@ons– Basedonhierarchicalaffinitygroups
• Checkpoint-con@nue– Automa@ccon@nuousasynchronouscheckpoin@ng
• Collabora@verun@mes– underdevelopment
• Inspector/executor– Forcomputa@on
• Demand-drivenexecu@on• Specula@veexecu@on
ZoranBudimlić
KathKnobe
![Page 38: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/38.jpg)
Isola@on:Tuningfromdomainspec
• Cri@calforsorwareengineeringforexascale• Cri@calforeaseoftuning• Justdataandcontroldependences– Noarbitraryconstraints– Exceptforgrain…
• CnCdomainspecistotallyflexiblewrttuning:– Moreapproachesinmindnow– Newarchitectures– Newtuninggoals– Newtuningapproaches
![Page 39: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/39.jpg)
Tuninggrainsize:Hierarchy
• Eachcomputa@onstepanddataitemcanbedecomposed
• Hierarchyaddsmoretuningflexibility– Choosethebesthierarchy – Choosethebestgrain– Differenttuningscanbeappliedatdifferentlevelsinthehierarchy
KathKnobeNickVrvilo
![Page 40: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/40.jpg)
Conclusion
• CnCisadependencelanguage– Controlanddatadependences– Explicit,dis@nctandatthesamelevel
• Tuningisseparate– NoCnC-specifictuning
• Thisisola@onmakestuningeasierandmoreflexible
![Page 41: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/41.jpg)
DARPA:UHPCDOE:X-StackCambridgeResearchLabDEC/Compaq/HP/Intel
CarlOffner,AlexNelsonIntel
FrankSchlimbach,JamesBrodman,MarkHampton,GeoffLowney,VincentCave,KamalSharma,X-StackTGteam
RiceVivekSarkar,ZoranBudimlić,MikeBurke,SanjayChauerjee,NickVrvilo,Mar@nKong,TiagoCogumbreiro
ReservoirRichLethinBenoitMeister
UCIrvineAparnaChandramowlishwaran(Intelintern)
GoogleAlinaSbirlea(Rice)DragosSbirlea(Rice)
UCSDLauraCarringtonPietroCicox
GaTechRichVuducHasnainMandviwala(Inelintern)KishoreRamachandran
IndianaRyanNewton(Intel)
PurdueMilindKulkarniChenyangLiu
PNNLJohnFeoEllenPorter
FacebookNicolasVasilache(Reservoir)
MicronKyleWheeler(xxNL)
DreamworksMar@nWau
TwoSigmaShamsImam(Rice)SagnakTasirlar(Rice
Thanksto…
![Page 42: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/42.jpg)
• IntelCnConIntel’sWhatIfsite
hup://sorware.intel.com/en-us/ar@cles/intel-concurrent-collec@ons-for-cc
Availableopensourcehups://icnc.github.io
• Ricehups://wiki.rice.edu/confluence/display/HABANERO/CNC
• DiscussCnCrelatedtopicsNewapps,op@miza@ons,run@me,tuning,…[email protected],[email protected],[email protected]
• CnC’16workshopSept27-28,2016Co-locatedwithLCPC’16inRochester,NYhups://cncworkshop2016.github.io/
![Page 43: CnC: A Dependence Programming Model](https://reader031.vdocuments.mx/reader031/viewer/2022012409/616a443711a7b741a350a1ed/html5/thumbnails/43.jpg)
Thankyou!