sc17 poster draftoptout.csc.ncsu.edu/~mueller/ftp/pub/mueller/papers/sc17poster2.pdf · title:...
TRANSCRIPT
Exploring Use-cases for Non-Volatile Memories in support of HPC ResilienceOnkarPatil,SaurabhHukerikar,FrankMueller,ChristianEngelmann
Dept.ofComputerScience,NorthCarolinaStateUniversity,ComputerScienceandMathematicsDivision,OakRidgeNationalLaboratory
MOTIVATION• Exaflop Computersà largenumberofcompute+memorydevices+differentformsofinterconnects+coolingandpowerequipmentà CloseProximity• Manufacturingprocessesusedtomakethesedevicesarenotfoolproof
• Lowerdurabilityandreliabilityofthedevices.• Frequencyofdevicefailuresanddatacorruptions↑à effectivenessandutility↓
• FutureApplicationsneedtobemoreresilientwhilethey,• Maintainabalancebetweenperformanceandpowerconsumption• Minimizetrade-offs
• Non-volatilememory(NVM)technologiesà enablememorydevicesthatcanmaintainstateofcomputationintheprimarymemoryarchitecture• Morepotentialinusingthesememorydevicesasspecializedhardware• DataRetentionà criticalinimprovingresilienceofanapplicationagainstcrashes• PersistentmemoryregionstoimproveHPCresiliencyà keyaspectofthisproject
PROBLEM STATEMENT
• Designstrategy• Enablecheckpointing atthedatastructurelevel• Somedatastructuresaremorecriticalthanothersatdifferentstagesoftheapplicationintermsoffailurerecovery• Reducethespaceandtimeoverheadconsiderablyincomparisontotraditionalcheckpointing methods• EasytouseAPIwithminimalcodechanges
NVM-basedMainMemory
APPROACHAPPLICATION
STATICDATASTRUCTURES
DYNAMICDATASTRUCTURES
DRAM NVM
APPLICATIONSTATICDATASTRUCTURES
DYNAMICDATASTRUCTURES
DRAM NVM
APPLICATIONSTATICDATASTRUCTURES
DYNAMICDATASTRUCTURES
DRAM NVM
Application-directedCheckpointing DataVersioningvoidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_only_alloc(size);…pmem_cpy_free(a);…}
voidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_cpy_alloc(size);…pmem_cpy_update(a);…pmem_cpy_free(a);…}
voidmain(){int *a,size;pmem_cpy_init();…a=(int *)pmem_ver_alloc(size);…pmem_cpy_update(a);…pmem_cpy_free(a);…}
• ExperimentationSetup• 16-nodeclusterwithDualsocket,Quad-CoreAMDOpteron,128GBDRAMmemory,IntelSSDfrom100GBto256GB• DGEMMbenchmarkoftheHPCCbenchmarksuite• Testedfor4,8and16-nodeconfigurationsforamatrixsizesof1000,2000and3000elements
RESULTS
0.001
0.01
0.1
1
10
4 8 16
GFLO
PS
Nodes
GFLOPSinnodescalingforStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
0.001
0.01
0.1
1
10
4 8 16
GFLO
PS
Nodes
GFLOPSinnodescalingforSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
• DRAMonlyallocationandNVM-basedmainmemoryperformbetterthanApplication-directedCheckpointing andDataVersioningpartlyduetoaninefficientlookupalgorithm
• TheperformanceisconsistentforbothSinglematrixmultiplicationandmultiplematrixmultiplicationoperations
• Allmodesperformsimilarandconsistentlywhenscaledbynodesizeorproblemsize
0.01
0.1
1
10
100
4 8 16
Time(sec)
Nodes
ExecutiontimesinnodescalingforStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
0.01
0.1
1
10
100
4 8 16
Time(sec)
Nodes
ExecutiontimesinnodescalingforSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
0.0001
0.001
0.01
0.1
1
10
1000 2000 3000
GFLO
PS
No.ofelements
GFLOPSforproblemsizescalinginStarDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
0.0001
0.001
0.01
0.1
1
10
1000 2000 3000
GFLO
PS
No.ofelements
GFLOPSforproblemsizescalinginSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
• Theexecutiontimeincreasesexponentiallywhenusingtwotypesofmemoryinsteadofone
0.01
0.1
1
10
100
1000
10000
1000 2000 3000
Time(sec)
No.ofelements
ExecutiontimeforproblemsizescalinginStarDGEMMDRAMPMEM_ONLYPMEM_CPYPMEM_VER
0.01
0.1
1
10
100
1000
10000
1000 2000 3000
Time(sec)
No.ofelements
ExecutiontimeforproblemsizescalinginSingleDGEMM DRAMPMEM_ONLYPMEM_CPYPMEM_VER
• Developthememoryusagemodestomakethemmoreefficientandmaintaincompletesystemstate• Minimaloverhead• Supportmorecomplexapplications
• Developlightweightrecoverymechanismstoworkwiththecheckpointingschemes• Reducedowntimeandrollbacktime
• Combinethemwithintelligentpoliciesthatcanhelpbuildresilientstaticanddynamicruntimesystem
FUTURE WORK
• Non-volatilememorydevicescanbeusedasspecializedhardwareforimprovingtheresilienceofthesystemandwedemonstratedthreepotentialmemoryusagemodelsthatshowconsistentperformanceforcomputeintensiveworkloads
• REFERENCES• Hukerikar,Saurabh,andChristianEngelmann."ResilienceDesignPatterns-AStructuredApproachtoResilienceatExtreme
Scale." arXiv preprintarXiv:1611.02717 (2016).• Mittal,Sparsh,andJeffreyS.Vetter."Asurveyofsoftwaretechniquesforusingnon-volatilememoriesforstorageandmain
memorysystems." IEEETransactionsonParallelandDistributedSystems 27.5(2016):1537-1550.• Hsu,TerryChing-Hsiang,etal."NVthreads:PracticalPersistenceforMulti-threadedApplications." ProceedingsoftheTwelfth
EuropeanConferenceonComputerSystems.ACM,2017.• Liu,Qingrui,etal."Compiler-directedlightweightcheckpointing forfine-grainedguaranteedsofterrorrecovery." High
PerformanceComputing,Networking,StorageandAnalysis,SC16:InternationalConferencefor.IEEE,2016.• Yang,Shuo,etal."Algorithm-DirectedCrashConsistenceinNon-VolatileMemoryforHPC." arXiv preprint
arXiv:1705.05541 (2017).• Wong,Daniel,G.S.Lloyd,andM.B.Gokhale. Amemory-mappedapproachtocheckpointing.No.LLNL-TR-635611.Lawrence
LivermoreNationalLaboratory(LLNL),Livermore,CA,2013.• Rezaei,Arash. FaultResilienceforNextGenerationHPCSystems.NorthCarolinaStateUniversity,2016.• ThisworkwassponsoredbytheU.S.DepartmentofEnergy'sOfficeofAdvancedScientificComputingResearch.This
manuscripthasbeenco-authoredbyUT-Battelle,LLCunderContractNo.DE-AC05-00OR22725withtheU.S.DepartmentofEnergy.TheUnitedStatesGovernmentretainsandthepublisher,byacceptingthearticleforpublication,acknowledgesthattheUnitedStatesGovernmentretainsanon-exclusive,paid-up,irrevocable,world-widelicensetopublishorreproducethepublishedformofthismanuscript,orallowotherstodoso,forUnitedStatesGovernmentpurposes.TheDepartmentofEnergywillprovidepublicaccesstotheseresultsoffederallysponsoredresearchinaccordancewiththeDOEPublicAccessPlan(http://energy.gov/downloads/doe-public-access-plan).
CONCLUSION