kent milfeld tacc may 28, 2009
DESCRIPTION
Profiling. Kent Milfeld TACC May 28, 2009. Outline. Profiling MPI performance with MPIp Tau gprof Example: matmult. MPIp. Scalable Profiling library for MPI apps Lightweight Collects statistics of MPI functions Communicates only during report generation Less data than tracing tools. - PowerPoint PPT PresentationTRANSCRIPT
Profiling
Kent Milfeld
TACCMay 28, 2009
Outline
• Profiling MPI performance with MPIp• Tau• gprof• Example: matmult
MPIp
• Scalable Profiling library for MPI apps• Lightweight• Collects statistics of MPI functions• Communicates only during report generation • Less data than tracing tools
http://mpip.sourceforge.net
Usage, Instrumentation, Analysis• How to use
– No recompiling required!– Profiling gathered in MPI profiling layer
• Link static library before default MPI libraries -g -L${TACC_MPIP_LIB} -lmpiP -lbfd -liberty -lintl –lm
• What to analyze– Overview of time spent in MPI communication during the
application run– Aggregate time for individual MPI call
Control• External Control: Set MPIP environment variable (threshold,
callsite depth)– setenv MPIP ‘-t 10 –k 2’– export MPIP= ‘-t 10 –k 2’
• Use MPI_Pcontrol(#) to limit profiling to specific code blocks
Pcontrol Arg Behavior
0 Disable profiling.
1 Enable Profiling.
2 Reset all callsite data.
3 Generate verbose report.
4 Generate concise report.
MPI_Pcontrol(2);MPI_Pcontrol(1); MPI_Abc(…);MPI_Pcontrol(0);
call MPI_Pcontrol(2)call MPI_Pcontrol(1) call MPI_Abc(…)call MPI_Pcontrol(0)
F90
C
• Untar the mpip tar file. % tar -xvf ~train00/mpip.tar; cd mpip• Read the “Instructions” file– setup Env. with sourceme-files% source sourceme.csh (C-type shells)% source sourceme.sh (Bash-type shells)
Example
module purge {Resets to default modules.}module load TACCmodule unload mvapich {Change compiler/MPI to intel/mvapich.} module swap pgi intelmodule load mvapich
module load mpiP {Load up mpiP.}
set path=($path /share/home/00692/train00/mvapich1_intel/Mpipview_2.02/bin){Sets PATH with mpipview dir. – separate software from mpiP.)}
• Compile either the matmultc.c or matmultf.f90 code.
% make matmultfOr
% make matmultc
• Uncomment “ibrun matmultc” or “ibrun matmultf” in the job file.
• Submit job.% qsub job
• Results in:<exec_name><unique number>.mpiP
Example (cont.)
Example *.mpiP Output
• MPI-Time: wall-clock time for all MPI calls
• MPI callsites
Output (cont.)
• Message size
• Aggregate time
e.g. % mpipview matmultf.32.6675.1.mpiPNote: list selection.
mpipview
Indexed
Call Sites
Statistics
Message Sizes
Tau Outline
• General– Measurements– Instrumentation & Control– Example: matmult
• Profiling and Tracing– Event Tracing– Steps for Performance Evaluation– Tau Architecture
• A look at a task-parallel MxM Implementation• Paraprof Interface
15
General
• Tuning and Analysis Utilities (11+ year project effort)
www.cs.uoregon.edu/research/paracomp/tau/
• Performance system framework for parallel, shared & distributed memory systems
• Targets a general complex system computation model– Nodes / Contexts / Threads
• Integrated toolkit for performance instrumentation, measurement, analysis, and visualization
TAU = Profiler and Tracer + Hardware Counters + GUI + Database
16
Tau: Measurements• Parallel profiling
– Function-level, block (loop)-level, statement-level– Supports user-defined events– TAU parallel profile data stored during execution– Hardware counter values– Support for multiple counters– Support for callgraph and callpath profiling
• Tracing– All profile-level events– Inter-process communication events– Trace merging and format conversion
17
PDT is used to instrument your code.
Replace mpicc and mpif90 in make files with tau_f90.sh and tau_cc.sh
It is necessary to specify all the components that will be used in the instrumentation (mpi, openmp, profiling, counters [PAPI], etc. However, these come in a limited number of combinations.)
Combinations: First determine what you want to do (profiling, PAPI counters, tracing, etc.) and the programming paradigm (mpi, openmp), and the compiler. PDT is a required component:
Tau: Instrumentation
PAPICallpath
…
MPIOMP
…
intelpgignu
PDTHand-code
Collectors Compiler:ParallelParadigmInstrumentation
18
You can view the available combinations(alias tauTypes 'ls -C1 $TAU | grep Makefile ' ).
Your selected combination is made known to the compiler wrapper through the TAU_MAKEFILE environment variable.
E.g. the PDT instrumention (pdt) for the Intel compiler (icpc) for MPI (mpi) is set with this command:
setenv TAU_MAKEFILE /…/Makefile.tau-icpc-mpi-pdt
Other run-time and instrumentation options are set through TAU_OPTIONS. For verbose:
setenv TAU_OPTIONS ‘-optVerbose’
Tau: Instrumentation
19
% tar –xvf ~train00/tau.tar
% cd tau READ the Instructions file % source sourceme.csh or % source sourceme.sh create env. (modules and TAU_MAKEFILE)
% make matmultf create executable(s)or
% make matmultc
% qsub job submit job (edit and uncomment ibrun line)
% paraprof (for GUI) Analyze performance data:
Tau Example
20
Definitions – Profiling• Profiling
– Recording of summary information during execution• inclusive, exclusive time, # calls, hardware statistics, …
– Reflects performance behavior of program entities• functions, loops, basic blocks• user-defined “semantic” entities
– Very good for low-cost performance assessment– Helps to expose performance bottlenecks and hotspots– Implemented through
• sampling: periodic OS interrupts or hardware counter traps• instrumentation: direct insertion of measurement code
21
Definitions – Tracing
TracingRecording of information about significant points (events)
during program executionentering/exiting code region (function, loop, block, …) thread/process interactions (e.g., send/receive message)
Save information in event record timestampCPU identifier, thread identifierEvent type and event-specific information
Event trace is a time-sequenced stream of event records Can be used to reconstruct dynamic program behaviorTypically requires code instrumentation
22
Event Tracing: Instrumentation, Monitor, Trace
1 master
2 worker
3 ...
void worker { trace(ENTER, 2);
... recv(A, tag, buf); trace(RECV, A);
... trace(EXIT, 2);
}
void master { trace(ENTER, 1);
... trace(SEND, B);
send(B, tag, buf); ...
trace(EXIT, 1);}
MONITOR58 A ENTER 1
60 B ENTER 2
62 A SEND B
64 A EXIT 1
68 B RECV A
...
69 B EXIT 2
...
CPU A:
CPU B:
Event definition
timestamp
23
Event Tracing: “Timeline” Visualization
1 master
2 worker
3 ...
58 A ENTER 1
60 B ENTER 2
62 A SEND B
64 A EXIT 1
68 B RECV A
...
69 B EXIT 2
...
mainmasterworker
58 60 62 64 66 68 70
B
A
24
Steps of Performance Evaluation
Collect basic routine-level timing profile to determine where most time is being spent
Collect routine-level hardware counter data to determine types of performance problems
Collect callpath profiles to determine sequence of events causing performance problems
Conduct finer-grained profiling and/or tracing to pinpoint performance bottlenecksLoop-level profiling with hardware countersTracing of communication operations
25
TAU Performance System Architecture
26
Overview of Matmult: C = A x B
CreateA & B
A BC = x
Order NP Tasks
MASTER Worker
Receive B
SendRow of A
Multiply row a x B
Send Back row of C
Receive a
ReceiveRow of C
Send B
jj x=
27
Preparation of Matmult: C = A x B
PE 0
CreateA
CreateB
GenerateA & B
BroadcastB to All
by columns
PE 0 PE x
MPI_Bcast( b(1,i)…
loop over i (i=1n)
PE 0
A BC = x
Order NP Tasks
MASTER
28
Master Ops of Matmult: C = A x B
Master (0) sends rows 1 through (p-1) to
slaves (1p-1) receives
1
p-1
PE 0 PE 1 -- p-1
MPI_Send(arow … i,i
loop over i (i=1p-1)
destinationtag
Master (0) receives rows 1 through (n) from
Slaves.
1
n
PE 0 PE 1 -- p-1
MPI_Recv(crow … ANY,k
loop over i (i=1n)
source,tag
MPI_Send(arow …idle,jdest,tag
A BC = x
Order NP Tasks
MASTER
29
Master Ops of Matmult: C = A x B
Slave(any) sends row j of C to master, PE 0
PE 0 MPI_Send( crow … j
jrow j
Slaves multiply all Columns of B into A (row i) to form row i of Matrix C
x=
Pick up broadcast of B columns from PE 0
MPI_Recv( arow …ANY,j
loop over i (i=1n)
Matrix * Vector
row j
A BC = x
Order NP Tasks
Worker
Slave receives anyA row from PE 0 j
j
30
Paraprof and Pprof
• Execute application and analyze performance data:• % qsub job
– Look for files: profile.<task_no>.
– With Multiple counters, look for directories for each counter.
• % pprof (for text based profile display)• % paraprof (for GUI)
– pprof and paraprof will discover files/directories.
– paraprof runs on PCs,Files/Directories can be downloaded to laptop and analyzed there.
31
Tau Paraprof Overview
HPMToolkit
MpiP
TAU
Raw files
PerfDMFmanaged(database)
Metadata
Application
Experiment
Trial
32
Tau Paraprof Manager WindowProvides Machine Details
Organizes Runs as: Applications, Experiments and Trials.
33
Routine Time Experiment
Profile Information is in “GET_TIME_OF_DAY” metric
Mean and Standard Deviation Statistics given.
34
Multiply_Matrices Routine Results
not from same run
Function Data Window gives a closer look at a single function:
35
Float Point OPS trial
Hardware Counters provide Floating Point Operations (Function Data view).
36
L1 Data Cache Miss trial
Hardware Counters provide L1 Cache Miss Operations.
37
Call Path
Call Graph Paths (Must select through “thread” menu.)
38
Call Path
TAU_MAKEFILE = …Makefile.tau-callpath-icpc-mpi-pdt
39
Derived MetricsSelect Argument 1 (green ball); Select Argument 2 (green ball);
Select Operation; then Apply. Derived Metric will appear as a new trial.
40
DerivedMetrics
Be careful even thoughratios are constant, coresmay do different amounts
of work/operations per call.
Since FP/Missratios are
constant– mustbe memory
access problem.
GPROF
GPROF is the GNU Project PROFiler. gnu.org/software/binutils/
•Requires recompilation of the code.
•Compiler options and libraries provide wrappers for each routine call, and periodic sampling of the program.
•A default gmon.out file is produced with the function call information.
•GPROF links the symbol list in the executable with the data in gmon.out.
Types of Profiles
• Flat Profile– CPU time spend in each function (self and cumulative) – Number of times a function is called– Useful to identify most expensive routines
• Call Graph– Number of times a function was called by other functions– Number of times a function called other functions– Useful to identify function relations– Suggests places where function calls could be eliminated
• Annotated Source– Indicates number of times a line was executed
Profiling with gprofUse the -pg flag during compilation:% gcc -g -pg srcFile.c% icc -g -pg srcFile.c% pgcc -g -pg srcFile.c
Run the executable. An output file gmon.out will be generated with the profiling information.
Execute gprof and redirect the output to a file:% gprof exeFile gmon.out> profile.txt% gprof -l exeFile gmon.out> profile_line.txt% gprof -A exeFile gmon.out>profile_anotated.txt
gprof example
Move into the profiling examples directory:
% cd profile/
Compile matvecop with the profiling flag:
% gcc -g -pg -lm matvecop.c {@ ~train00/matvecop.c}
Run the executable to generate the gmon.out file:
% ./a.out
Run the profiler and redirect output to a file:
% gprof a.out gmon.out > profile.txt
Open the profile file and study the instructions.
Visual Call Graph
mainmain
matSqrtmatSqrt
vecSqrtvecSqrt
matCubematCube vecCubevecCubesysSqrtsysSqrt sysCubesysCube
Flat profile
In the flat profile we can identify the most expensive parts of the code (in this case, the calls to matSqrt, matCube, and sysCube).
% cumulative self self total
time seconds seconds calls s/call s/call name
50.00 2.47 2.47 2 1.24 1.24 matSqrt
24.70 3.69 1.22 1 1.22 1.22 matCube
24.70 4.91 1.22 1 1.22 1.22 sysCube
0.61 4.94 0.03 1 0.03 4.94 main
0.00 4.94 0.00 2 0.00 0.00 vecSqrt
0.00 4.94 0.00 1 0.00 1.24 sysSqrt
0.00 4.94 0.00 1 0.00 0.00 vecCube
Call Graph Profileindex % time self children called name
0.00 0.00 1/1 <hicore> (8)
[1] 100.0 0.03 4.91 1 main [1]
0.00 1.24 1/1 sysSqrt [3]
1.24 0.00 1/2 matSqrt [2]
1.22 0.00 1/1 sysCube [5]
1.22 0.00 1/1 matCube [4]
0.00 0.00 1/2 vecSqrt [6]
0.00 0.00 1/1 vecCube [7]
-----------------------------------------------
1.24 0.00 1/2 main [1]
1.24 0.00 1/2 sysSqrt [3]
[2] 50.0 2.47 0.00 2 matSqrt [2]
-----------------------------------------------
0.00 1.24 1/1 main [1]
[3] 25.0 0.00 1.24 1 sysSqrt [3]
1.24 0.00 1/2 matSqrt [2]
0.00 0.00 1/2 vecSqrt [6]
-----------------------------------------------
Profiling dos and don’ts
DO
• Test every change you make
• Profile typical cases• Compile with
optimization flags• Test for scalability
DO NOT
• Assume a change will be an improvement
• Profile atypical cases• Profile ad infinitum
– Set yourself a goal or– Set yourself a time limit
PAPI Implementation
Tools
PAPI Low LevelPAPI High Level
Hardware Performance Counter
Operating System
Kernel Extension
PAPI Machine Dependent SubstrateMachine
SpecificLayer
PortableLayer
50
PAPI Performance Monitor
• Provides high level counters for events:– Floating point instructions/operations, – Total instructions and cycles– Cache accesses and misses– Translation Lookaside Buffer (TLB) counts– Branch instructions taken, predicted, mispredicted
• PAPI_flops routine for basic performance analysis– Wall and processor times– Total floating point operations and MFLOPS http://icl.cs.utk.edu/projects/papi
• Low level functions are thread-safe, high level are not
51
High-level API
• C interfacePAPI_start_countersPAPI_read_countersPAPI_stop_countersPAPI_accum_countersPAPI_num_countersPAPI_flips PAPI_ipc
• Fortran interfacePAPIF_start_countersPAPIF_read_countersPAPIF_stop_countersPAPIF_accum_countersPAPIF_num_countersPAPIF_flipsPAPIF_ipc
52
Instrumenting Code:1.) Specify “events” in an array.2.) Initialize library.3.) Start counting events.4.) Stop counting (and accumulate).