porting an mpi application to hybrid mpi+openmp with reveal tool on shaheen ii

KAUSTSupercompu.ngLaboratoryPor.nganMPIapplica.ontohybridMPI+OpenMPwithRevealtoolonShaheenII

GeorgeMarkomanolisComputa.onalScien.stJune23th,2016

Outline

KAUST King Abdullah University of Science and Technology 2

❖  Introduction

❖  Test case

❖  Reveal

Introduc.on-ComponentsofCrayPat


❖  Module perftools-base

•  pat_build – Instruments the program to be analyzed •  pat_report – Generates text reports from the performance data

captured during program execution and exports data for use in other programs.

•  Cray Apprentice2 – A graphical analysis tool that can be used to visualize and explore the performance data captured during program, execution

•  Reveal – A graphical source code analysis tool that can be used to correlate performance analysis data with annotated source code listings, to identify key opportunities for optimization (it works only with Cray compiler)

Studyingcase


❖  Application from seismic group related to acoustic wave

solver •  Why this application? A user asked for it •  MPI application •  Test on 3 nodes with totally 96 cores on

Shaheen II

Prepareforthetutorial


•  Connect to Shaheen II and copy the material: •  ssh –X [email protected]

•  cp /scratch/tmp/model_reveal.tgz .

•  tar zxvf model_reveal.tgz

•  cd model_reveal

•  Reservation name: k1056_141

Reveal

A tool to port your application to OpenMP programming model


Reveal


❖ Reveal is Cray’s next-generation integrated performance analysis and code optimization tool.

•  Source code navigation using whole program

analysis (data provided by the Cray compilation environment only)

•  Coupling with performance data collected during execution by CrayPAT. Understand which high level serial loops could benefit from parallelism.

•  Enhanced loop mark listing functionality. •  Dependency information for targeted loops •  Assist users optimize code by providing variable

scoping feedback and suggested compile directives.

PrepareforReveal


❖  Load Perftools •  module unload darshan •  module load perftools-base/6.3.2 •  module load perftools/6.3.2

❖  Execute the MPI version •  cd model_reveal •  make clean •  make •  In the submit.sh file changed to your account number and submit the

job §  sbatch submit.sh

•  tail -n 10 testdata.XXX.err §  1m46.361s

Reservation: k1056_141

Preparetheapplica.onforReveal


❖  Compile the version for Reveal tool •  make clean –f Makefile_reveal •  In the Makefile_reveal file

§  $(CC) -h profile_generate -hpl=data.pl -h noomp $< -o $@ $(CFLAGS)

§  ${CC} -h profile_generate -hpl=data.pl -h noomp -c $< CrayData.c §  Reveal needs the object of the files, so you need to modify the

Makefile if needed •  make –f Makefile_reveal •  The folder data.pl is created in the folder •  Instrument your application

§  pat_build –w CrayData.exe §  New executable is called CrayData.exe+pat, replace it to submit.sh

SubmitthejobforRevealtool


❖  Submit your job script and do not forget the reservation name (--reservation=…)

•  sbatch submit.sh

❖  A performance file (extension .xf) is created, if not something was wrong in the previous steps

❖  Generate the report and the ap2 file •  pat_report -o report.txt CrayData.exe+pat+58072-37t.xf

❖  Execute Reveal •  reveal data.pl CrayData.exe+pat+58072-37t.ap2

Reveal–LoopPerformance


Reveal–Scoping


Reveal–Programview


Reveal–Func.onView


Reveal–ArrayView


Reveal–CompilerMessages


Reveal–LoopPerformance


Reveal–ScopingTool


Reveal–ScopingResults


Reveal–OpenMPpragmas


Reveal–InsertedOpenMPpragmas


CleanthecodefromunresolvedissuesandobserveOpenMPpragmas


❖  vim CrayData.c ❖  Remove the lines with unresolved, only if you are sure.

#pragma omp parallel for default(none) \ private (i1,i2,u) \ shared (nxpad,nzpad)

#pragma omp parallel for default(none) \ private (ix,ib,ibz) \ shared (nxpad,nb,nzpad,bndr,p0) \ lastprivate (w)

CheckanOpenMPpragmaanditsvalida.on


#pragma omp parallel for default(none) private (ix,ib,ibz) \ shared (nxpad,nb,nzpad,bndr,p0) \ lastprivate (w) for(ix=0; ix<nxpad; ix++) {

for(ib=0; ib<nb; ib++) { w = bndr[nb-ib-1]; ibz = nzpad-ib-1;

p0[ix][ib ] *= w; /* top sponge */ p0[ix][ibz] *= w; /* bottom sponge */ } } for(ib=0; ib<nb; ib++) { ibx = nxpad-ib-1; for(iz=0; iz<nzpad; iz++) { p0[ib ][iz] *= w; /* left sponge */

p0[ibx][iz] *= w; /* right sponge */ } }

Cleanthecodefromunresolvedissues,compileandrun


❖  vim CrayData.c ❖  Remove the lines with unresolved if you are sure. ❖  Compile your application with MPI and OpenMP

•  make –f Makefile_omp •  The new executable is called CrayData_omp.exe •  Comment the active srun line in the submit.sh and uncomment

the next srun call. •  Uncomment also the line with OMP_NUM_THREADS=2 •  Now, we will execute the application with 48 MPI processes

(ntasks) and 2 threads per MPI process (cpus-per-task) •  srun --ntasks=48 --ntasks-per-node=16 --ntasks-per-socket=8 --

hint=nomultithread --cpus-per-task=2 ./CrayData_omp.exe

Differentcasesandresults


❖  Results for 2 threads •  Change according:

§  export OMP_NUM_THREADS=2 §  srun –ntasks=48 --ntasks-per-node=16 --ntasks-per-

socket=8 --hint=nomultithread --cpus-per-task=2 ./CrayData_omp.exe

•  51.211s (2.86X)

❖  Results 4 threads •  Change according:

§  export OMP_NUM_THREADS=4 §  srun --ntasks=24 --ntasks-per-node=8 --ntasks-per-socket=4

--hint=nomultithread --cpus-per-task=4 ./CrayData_omp.exe •  24.815s (5.9X)

Differentcasesandresults


❖  Results 8 threads •  12.222s (11.98X)

❖  Results 16 threads •  Change according:

§  export OMP_NUM_THREADS=16

§  srun --ntasks=6 --ntasks-per-node=2 --ntasks-per-socket=1 --hint=nomultithread --cpus-per-task=16 ./CrayData_omp.exe

•  8.895s (16.45X)

Theoriginalversionwasimproved19.19.mes


170.67

106.36

8.8950

20406080

100120140160180

Originalversion Op.mizedMPIversion

MPI+OpenMP

Time(in

sec.)

Execu.on.me

Valida.on


Original version Optimized MPI+OpenMP

Summary


❖  Reveal is an easy to use tool

❖  The user should be careful though, give notice to compiler messages

❖  You can have great speedup with this tool

❖  We need to investigate more complicated applications

KAUST Supercomputing Laboratory
