performance measurement and modeling of component

Performance Measurement and Modeling of Component Applications in a HighPerformance Computing Environment : A Case Study

J. Ray, N. Trebon, R. C. ArmstrongSandia National Laboratories,

Livermore, CA 94551�jairay,ndtrebo,rob�@sandia.gov

S. Shende and A. MalonyDept. of Computer Science,

University of Oregon, Eugene, OR 97403�sameer,malony�@cs.uoregon.edu

Abstract

We present a case study of performance measurementand modeling of a CCA (Common Component Architecture)component-based application in a high performance com-puting environment. Component-based HPC applicationsallow the possibility of creating component-level perfor-mance models and synthesizing them into application per-formance models. However, they impose the restriction thatperformance measurement/monitoring needs to be done ina non-intrusive manner and at a fairly coarse-grained level.We propose a performance measurement infrastructure forHPC based loosely on recent work done for Grid environ-ments. A prototypical implementation of the infrastructureis used to collect data for three components in a scientificapplication and construct their performance models. Bothcomputational and message-passing performance are ad-dressed. 1

1 Introduction

The performance of scientific simulations in high per-formance computing (HPC) environments is fundamentallygoverned by the (effective) processing speed of the indi-vidual CPUs and the time spent in interprocessor commu-nications. The effective processing speed is primarily de-termined by the performance of the cache (in cache-basedRISC and CISC processors) and much effort is devotedto preserving data locality. Interprocessor communicationcosts determine the load-balancing and scalability charac-teristics of codes and a multitude of software and algorith-mic strategies (combining communication steps, minimiz-ing/combining global reductions and barriers, overlappingcommunications with computations, etc.) are employed toreduce them.

1Presented at the 18th International Parallel and Distributed Process-ing Symposium, Santa Fe, NM, USA, April ��

�� , 2004.

The discipline of performance measurement has pro-vided us with tools and techniques to gauge the interac-tions of scientific applications with the execution platform.These take the form of high precision timers which reportthe time taken to execute sections of the code and variouscounters which report on the behavior of various compo-nents of the hardware as the code executes[2, 3]. In a par-allel environment these tools track and report on the size,frequency, source, destination and the time spent in passingmessages between processors [19]. This information canthen be used to synthesize a performance model of the ap-plication on the given platform - in some cases, these mod-els have even served in a predictive capacity [11, 12].

In order to manage the growing complexity of scien-tific simulation codes, there has been an effort to introducecomponent-based software methodology in HPC environ-ments. Popular component models like Java Beans [9] andCORBA [1] are largely unsuitable for HPC [5] and a newlight-weight model, called the Common Component Archi-tecture (CCA) [6], was proposed. The principal motiva-tions behind the CCA are to promote code reuse and in-terdisciplinary collaboration in the high performance com-puting community. The component model consists of mod-ularized components with standard, well-defined interfaces.Since components communicate through these interfaces,program modification is simplified to modifying a singlecomponent or switching in a similar component without af-fecting the rest of the application. To build a CCA applica-tion, an application developer simply composes together aset of components using a CCA-compliant framework. De-tails regarding the flexibility, performance and design char-acteristics of CCA applications can be found in [13].

CCA component-based applications are composed outof standalone components at runtime, an injudicious se-lection of which can result in a correct but sub-optimalcomponent assembly. It thus becomes imperative to beable to classify the performance characteristics and require-ments of each implementation of a component and to havea generalized means of synthesizing a composite perfor-

0-7695-2132-0/04/$17.00 (C) 2004 IEEE

Proceedings of the 18th International Parallel and Distributed Processing Symposium (IPDPS’04)

mance model to judge the optimality of a component as-sembly. Further component-based software is seldom usedexclusively by the authors of the components themselvesand manual instrumentation of the code is impossible. Ex-haustive automatic instrumentation of an executable wherea binary is rewritten or instrumented at runtime [18] toohas little meaning. Consequently, a non-intrusive strategywhere each component is monitored to collect executiontime, the hardware characteristics and relevant inputs (likethe size of arrays) that affect the collected performance datais clearly indicated. These data then need to be synthesizedinto individual component performance models. Also, sincethe containing CCA framework creates, configures and as-sembles components, it possesses the global understandingof how the components are networked into an application.This information, coupled with the individual components’performance models, can be used to synthesize a predic-tive performance model of the entire application. In thiswork, we will attempt to create a small set of componentsto assemble a non-intrusive performance measurement andmodeling infrastructure. This infrastructure will then beused to monitor an existing CCA-component based scien-tific simulation code (assembly of CCA components) andconstruct performance models for the components. It is as-sumed the none of the scientific components will be modi-fied to assist in performance measurement nor do they haveany special features (e.g. performance-related interfaces)that allow or enable the collection of performance-relateddata. While the material presented in this work is far fromrealizing the goal of synthesizing an application-level per-formance model and using it in a predictive capacity, it isessential that it be viewed as step toward realizing a com-pletely automated system for performance prediction – andhence optimization – of high performance component basedapplications.

2 Related Work

The three most widely-used component standards(CORBA [1], COM/DCOM [14], Java Beans [9]) are ill-suited to handle high performance scientific computing dueto a lack of support for efficient parallel communication,insufficient scientific data abstractions (e.g., complex num-bers), and/or limited language interoperability [6]. Thus,performance metrics developed for these environments areinadequate for HPC. This primarily arises from the verydifferent platforms that HPC and commercial componentbased applications run on - HPC is done almost exclusivelyon tightly-connected clusters of MPPs (massively parallelprocessors) or SMPs (Symmetric Multi-processors) whilecommercial codes often operate on LANs (Large Area Net-works) or WANs (Wide Area Networks).

However, despite the different semantics, several re-

search efforts in these standards offer viable strategies inmeasuring performance. A performance monitoring systemfor the Enterprise Java Beans standard is described in [16].For each component to be monitored, a proxy is created us-ing the same interface as the component. The proxy inter-cepts all method invocations and notifies a monitor com-ponent before forwarding the invocation to the component.The monitor handles the notifications and selects the datato present, either to a user or to another component (e.g., avisualizer component). The goal of this monitoring systemis to identify hot spots or components that do not scale well.

The Wabash tool [20, 21] is designed for pre-deploymenttesting and monitoring of distributed CORBA systems. Be-cause of the distributed nature, Wabash groups componentsinto regions based on the geographical location. An inter-ceptor is created in the same address space of each serverobject (i.e., a component that provides services) and man-ages all incoming and outgoing requests to the server. Amanager component is responsible for querying the inter-ceptor for data retrieval and event management.

In the work done by the Parallel Software Group at theImperial College of Science in London [10], the researchis focused on grid-based component computing. However,the performance is also measured through the use of prox-ies. Their performance system is designed to automaticallyselect the optimal implementation of the application basedon performance models and available resources. With �

components, each having �� implementations, there is a to-tal of ��

�� implementations to choose from. The per-

formance characteristics and a performance model for eachcomponent is constructed by the component developer andstored in the component repository. Their approach is touse the proxies to simulate an application in order to de-termine the call-path. This simulation skips the implemen-tation of the components by using the proxies. Once thecall-path is determined, a recursive composite performancemodel is created by examining the behavior of each methodcall in the call-path. In order to ensure that the compositemodel is implementation-independent, a variable is used inthe model whenever there is a reference to an implemen-tation. To evaluate the model, a specific implementation’sperformance model replaces the variables and the compos-ite model returns an estimated execution time or estimatedcost (based on some hardware resources model). The im-plementation with the lowest execution time or lowest costis then selected and a execution plan is created for the ap-plication.

3 Performance Measurements in HPC Com-ponent Environments

As indicated in Section 1, performance measurement andmodeling (PMM), in a component environment, will assist

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


in optimizing the component assembly. It is required thatPMM strategies (a) provide a coarse-grained performancemodel of the component and (b) be non-intrusive. The sim-plest approach, as reviewed in Section 2, is that of proxies,interposed between the caller and the called components,which intercept method calls and execute performance re-lated tasks.

In this section we provide a brief summary of the CCAenvironment for HPC, adapt the approaches in Section 2to HPC and address the issue of the minimal set of perfor-mance data required to construct component-level perfor-mance models.

3.1 The Common Component Architecture(CCA)

The CCA model uses the provides-uses design pattern.Components provide functionalities through interfaces thatthey export; they use other components’ functionalities viainterfaces. These interfaces are called Ports; thus a com-ponent has ProvidesPorts and UsesPorts. Components arepeers and are independent. They are created and exist insidea framework; this is where they register themselves, declaretheir UsesPorts and ProvidesPorts and connect with othercomponents.

CCAFFEINE [5] is the CCA framework we employ forour research. CCAFFEINE is a low latency frameworkfor scientific computations. Components can be written inmost languages within the framework; we develop most ofour components in C++. All CCAFFEINE components arederived from a data-less abstract class with one deferredmethod called setServices(Services *q). All componentsimplement the setServices method which is invoked by theframework at component creation and is used by the com-ponents to register themselves and their UsesPorts and Pro-videsPorts. Components also implement other data-less ab-stract classes, called Ports, to allow access to their standardfunctionalities. Every component is compiled into a sharedobject library that will be dynamically loaded at runtime.

A CCAFFEINE code can be assembled and run througha script or a Graphical User Interface (GUI). All compo-nents exist on the same processor and the same addressspace. Once components are instantiated and registeredwith the framework, the process of connecting ports is justthe movement of (pointers to) interfaces from the provid-ing to the using component. A method invocation on a Us-esPort thus incurs a virtual function call overhead beforethe actual implemented method is used. CCAFFEINE usesthe SCMD (Single Component Multiple Data) [5] modelof parallel computation. Identical frameworks, containingthe same components, are instantiated on all � processors.Parallelism is implemented by running the same compo-nent on all � processors and using MPI to communicate

between them. The framework adheres to the MPI-1 stan-dard ; dynamic process creation/deletion and a dynamicallysized parallel virtual machine are not supported. This min-imalist nature renders CCAFFEINE light, simple, fast, andvery unobtrusive to the components. Performance is left tothe component developer who is in the best position to de-termine the optimal algorithms and implementations for theproblem at hand.

3.2 Performance Measurement and Modeling

A CCA application is composed of components and thecomposite performance of a component assembly is deter-mined by the performance of the individual components aswell as the efficiency of their interaction. Thus, the per-formance of a component has to be considered in a certaincontext consisting of the problem being solved (e.g., a com-ponent may have to do two functions, one which requiressequential access and the other strided access of an array),the parameters/arguments being passed to a method (e.g.,length of an array) and the interaction between the callerand the callee (e.g., if a transformation of the data storageneeds to be done). If multiple implementations of a com-ponent exist (i.e., implementations which provide the samefunctionality) then within a given context, there will be anoptimal choice of implementation. This requires that perfor-mance models be available for all components and a meanssynthesize these into a model for the application exist.

Most scientific components intersperse compute inten-sive phases with message passing calls, which incur costsinversely proportional to the network speed. These callssometimes involve global reductions and barriers, result-ing in additional synchronization costs. For the purposes ofthis paper we will assume blocking communications wherecommunications and computations are not overlapped. Wewill ignore disk I/O in this study. Thus, in order that a per-formance model for a component may be constructed, werequire the following :

1. The total execution time spent in a method call. Thesemethods are those in the ProvidesPorts of a compo-nent.

2. The total time spent in message passing calls, as deter-mined by the total inclusive time spent in MPI duringa method invocation.

3. The difference between the above is the time spentin computation, a quantity sensitive to the cache-hitrate. We will record this quantity for the period of themethod call.

4. The input parameters that affect performance. Thesetypically involve the size of the data being passed to thecomponent and some measure of repetitive operations

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


that might need to be done (e.g., the number of times asmoother may be applied in a multigrid solution).

The first three requirements are traditional and may beobtained from publicly available tools [4]. The fourthrequires some knowledge of the algorithms being imple-mented, and is extracted by a proxy before the method in-vocation is forwarded to the component. We envisage thatproxies will be simple and preferably, amenable to auto-matic generation.

4 PMM Software Infrastructure

As stated in Section 3, performance measurement willbe done via proxies interposed between caller and calleecomponents. These proxies are expected to be lightweightand serve as a means of intercepting and forwarding methodcalls. The actual functionality of interacting with andrecording hardware characteristics will be kept in a sepa-rate component, as will the functionality of storing this datafor each invocation. Our performance system consists ofthree distinct component types: a TAU (Tuning and Anal-ysis Utilities) component, proxy components and a “Mas-termind” component. These components work together inorder to measure, compile and report the data back to theuser.

4.1 TAU Component

In order to measure performance in a high performancescientific environment, a component that can interact withthe system’s hardware as well as time desired events isneeded. For our performance measurement system, we usethe TAU component[19], which utilizes the TAU measure-ment library[4, 15]. The TAU component is accessed viaa MeasurementPort, which defines interfaces for timing,event management, timer control and measurement query.The timing interface provides a means to create, name, start,stop and group timers. It helps track performance data as-sociated with a code region by bracketing it with start andstop calls.

The TAU implementation of this generic performancecomponent interface supports both profiling and tracingmeasurement options. Profiling records aggregate inclusiveand exclusive wall-clock time, process virtual time, hard-ware performance metrics such as data cache misses andfloating point instructions executed, as well as a combi-nation of multiple performance metrics. The event inter-face helps track application and runtime system level atomicevents. For each event of a given name, the minimum, max-imum, mean, standard deviation and number of entries arerecorded. TAU relies on an external library such as PAPI[2] or PCL [3] to access low-level processor-specific hard-ware performance metrics and low latency timers. Timer

control is achieved through the control interface, which canenable and disable timers of a given group at runtime. Atruntime, a user can enable or disable all MPI timers via theirgroup identifier. The query interface provides a means forthe program to access a collection of performance metrics.In our performance system, the query interface is used toobtain the current values for the metrics being measured.The TAU library also dumps out summary profile files atprogram termination.

4.2 Proxies

For each component that the user wants to analyze, aproxy component is created. The proxy component sharesthe same interface as the actual component. When the ap-plication is composed and executed, the proxy is placed di-rectly “in front” of the actual component. Since the proxyimplements the same interface as the component, the proxyintercepts all of the method invocations for the component.In other words, the proxy uses and provides the same typesof ports that the actual component provides. In this man-ner, the proxy is able to snoop the method invocation onthe Provides Port, and then forward the method invocationto the component on the Uses Port. In addition, the proxyalso uses a Monitor port (provided by the Mastermind com-ponent; see below) to make measurements. If the methodis one that the user wants to measure, monitoring is startedbefore the method invocation is forwarded and stopped af-terward. When the monitoring is started, parameters thatinfluence the method’s performance are sent by the proxyto the Mastermind component. These parameters must beselected by someone with a knowledge of the algorithm im-plemented in the component. For example, for a routine thatperforms some simple processing on each index of an arrayof numbers, the performance parameter would most likelybe the size of the array. Creating a proxy from a compo-nent’s header file is relatively straight-forward and effort isunder way to automate it.

4.3 Mastermind

The Mastermind component is responsible for gather-ing, storing and reporting of the measurement data. Foreach method that is monitored, a record object is createdand stored by the Mastermind. The record object stores allthe measurement data for each of the invocations of a sin-gle routine. When monitoring is started via a call to theMastermind, the Mastermind passes the parameters to therecord object and tells the record to begin timing. To makea measurement, the TAU component is queried at the startand the end of each method invocation to turn on/off tim-ing and hardware counters, prior to recording the measure-ments. The MPI time is determined by the summation of

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


C1 C3

C2

C4

M TP1

P2

P3

P4

Figure 1. A simple application composed of 4components. C denotes a component, P de-notes a proxy and M and T denote an instanceof Mastermind and TauMeasurement compo-nents. The black lines denote port connec-tion between components and the dashedlines are the proxy-to-Mastermind port con-nections. The proxies, along with the Mas-termind, TauMeasurement and the dashed portconnections form the non-intrusive PMM in-frastructure “around” the original componentassembly.

the times of all the MPI routines. The single invocationmeasurements, along with the parameters, are stored in therecord. When a record object is destroyed, it outputs to afile all of the measurement data for each invocation that itstored.

Thus we envisage a non-intrusive performance measure-ment and modeling component infrastructure built aroundan existing scientific simulation component assembly. Fig-ure 1 shows a component assembly of four components witha PMM infrastructure of proxies, Mastermind and TAUbuilt around it. The infrastructure is non-intrusive and mod-ification of the scientific components was not required.

5 Case Study

We use the infrastructure described in Section 4 to mea-sure and model the performance of a component-based sci-entific simulation code. The code simulates the interac-tion of a shock wave with an interface between two gases.The scientific details are in [17]. The code employs Struc-tured Adaptive Mesh Refinement [8, 7] for solving a set ofPartial Differential Equations (PDE) called the Euler equa-tions. Briefly, the method consists of laying a relativelycoarse Cartesian mesh over a rectangular domain. Based onsome suitable metric, regions requiring further refinementare identified, the grid points flagged and collated into rect-angular children patches (also called “boxes”) on which adenser Cartesian mesh is imposed. The refinement factor

x

y

0 0.5 1 1.5 20

0.5

1

1.5

2

Patches

Level 1 Patch

Level 2 Patch

Figure 2. The density field plotted for a Mach1.5 shock interacting with an interface be-tween Air and Freon. The simulation was runon a 3-level grid hierarchy. Patches outlinedwith the thickest lines are the coarsest (Level0), those outlined with medium lines are onLevel 1 (refined once by a factor of 2) and theones outlined with the finest lines are twicerefined (Level 2).

between parent and child mesh is usually kept constant fora given problem. The process is done recursively, so thatone ultimately obtains a hierarchy of patches with differ-ent grid densities, with the finest patches overlaying a smallpart of the domain. The more accurate solution from thefinest meshes is periodically interpolated onto the coarserones. Typically, patches on the coarsest level are processedfirst, followed recursively by their children patches. Chil-dren patches are also processed a set number of times dur-ing each recursion.

Figure 2 shows a snapshot from the simulation. Level 0boxes are the coarse mesh patches, Level 1 patches are thosewhich have been refined once and Level 2 patches have beenrefined twice. The factor of refinement is 2 and the sequenceof processing is �� , where �� is theset of patches on level �. Patches can be of any size or aspectratio. This sequence is repeated multiple times.

Figure 8 shows the component version of the code. Onthe left is the ShockDriver, a component that orchestratesthe simulation. On its right is AMRMesh that manages thepatches. The RK2 component below it orchestrates the re-

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


cursive processing of patches. To its right are StatesCon-structor and EFMFlux which are invoked on a patch-by-patch basis. The invocations to StatesConstructor andEFMFlux include a data array (a different one for eachpatch) and an output array of the same size. Neither ofthese components involve message passing, most of whichis done by AMRMesh. We will attempt to model the perfor-mance of both StatesConstructor and EFMFlux and an-alyze the message passing costs of AMRMesh. We willalso analyze the performance of another component, Go-dunovFlux, which can be substituted for EFMFlux. Threeproxies, one each for StatesConstructor, GodunovFluxand EFMFlux were created and interposed between Invis-cidFlux and the component in question. A proxy was alsowritten for AMRMesh to capture message-passing costs.An instance each of Mastermind and TAUMeasurementcomponent were created for performance measurement andrecording.

The simulation was run on three processors of a clus-ter of dual 2.8 GHz Pentium Xeons with 512 kB caches.gcc version 3.2 was used for compiling with -O2 opti-mization. Table 1 shows where most of the time is spentin the component code. About 25% of the time is spentin MPI Waitsome() which is invoked from two meth-ods in AMRMesh - one that does “ghost-cell updates” onpatches (gets data from abutting, but off-processor patchesonto a patch) and the other that results in load-balancingand domain (re-) decomposition. The other methods, one inStatesConstructor and the other in EFMFlux are modeledbelow.

In Figure 3 we plot the execution times for each invo-cation of StatesConstructor, as collected during a simu-lation. The StatesConstructor component is invoked intwo modes, one which requires sequential and the otherwhich requires strided access of arrays to calculate X- andY- derivatives of a field respectively. Sequential and stridedaccess of a given array occurs in an interleaved manner, asdictated by the numerical algorithm. Both the times (perinvocation) are plotted. We see that for a given array per-invocation execution time is not constant, especially forlarge arrays. The Y-derivative calculation (strided access)is expected to take longer for large arrays and this is seen inthe spread of timings. On the other hand, for small, largelycache-resident arrays, both the modes take roughly the sametime. As the arrays overflow the cache, the strided modebecomes more expensive and one sees a localization of tim-ings around two foci. The ratio of strided and sequentialaccess times varies from � � for small arrays to � � forlarge ones. Further, for larger arrays, one observes largescatters. Similar phenomena are also observed for both Go-dunovFlux and EFMFlux (not shown here).

Since the StatesConstructor is invoked to calculate theX- and Y-derivatives an equal number of times, we will con-

Array Size

Com

pute

Tim

e(m

icro

seco

nds)

50000 100000 150000

1.00E+04

2.00E+04

3.00E+04

4.00E+04

5.00E+04

6.00E+04

7.00E+04

8.00E+04

9.00E+04

1.00E+05

1.10E+05

1.20E+05

1.30E+05

Proc 0Proc 1Proc 2

Figure 3. Raw per invocation execution timefor the StatesConstructor component, col-lected during a simulation, plotted as a func-tion of the data array size. For a given ar-ray size, the larger timings correspond tothe calculation of Y-derivatives on the fieldsince it involves strided access of non-cache-resident arrays. Array sizes are the actualnumber of double precision numbers in thearray. The different symbols (�, Æ and�) rep-resent data from different processors (Proc �

in the legend) and similar trends are seen onall processors.

sider the average of the sequential and strided array process-ing times for modeling. However, we also include a stan-dard deviation in our analysis to track the variability intro-duced by the cache. It is expected that both the mean and thestandard deviation will be sensitive to the cache size. In Fig-ures 4 and 5 we plot the execution times for the StatesCon-structor and EFMFlux components. Regression analysiswas used to fit simple polynomial and power laws, whichare also plotted in the figures. The mean execution timescales linearly with the array size, once the cache effectshave been averaged out. Similar trends are seen the theanalysis for GodunovFlux. (not shown here). Note thatthese timings do not include the cost of the work done inthe proxies, since all the extraction and recording of param-eters is done outside the timers and counters that actuallymeasure the performance of a component. Further, theseinstrumentation related overheads are small and will not be

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


Array Size

Mea

n

Std

.Dev

iatio

n

102 103 104 105100

101

102

103

104

100

101

102

103

104MeanStd. Deviation

Figure 4. Average execution time (�) forStatesConstructor as a function of the arraysize. Since StatesConstructor has a dual modeof operation (sequential versus strided) andthe mean includes both, the standard devia-tion of is rather large. The performance modelis given in Eq. 1. The standard deviation (Æ)is plotted against the right Y-axis. All timingsare in microseconds. Solid lines are best-fitsto the data.

addressed in this paper.If �� and �� are the exe-

cution times (in microseconds) for StatesConstructor, Go-dunovFlux and EFMFlux and � the input array size, thebest-fit expressions for the three components are

��

��

�� (1)

The corresponding expressions for the standard deviations� are

��

��

��

��

�� (2)

We see that GodunovFlux is more expensive that EFM-Flux, especially for large arrays. Further, the variability in

Array Size

Mea

n

Std

.Dev

iatio

n

50000 100000 1500001

5001

10001

15001

20001

25001

500

1000

1500

2000

2500

3000

MeanStd. Deviation

Figure 5. Average execution time (�) for EFM-Flux as a function of the array size. SinceEFMFlux has a dual mode of operation (se-quential versus strided) and the mean in-cludes both, the standard deviation of israther large. The performance model is givenin Eq. 1. The standard deviation (Æ) is plot-ted against the right Y-axis. All timings are inmicroseconds. Solid lines are best-fits to thedata.

timings (i.e., standard deviation) for GodunovFlux increasewith � while its behavior is more complex for EFMFlux.While GodunovFlux is the preferred choice for scientists(it is more accurate), from a performance point of view,EFMFlux has better characteristics. This is an excellent ex-ample of a Quality of Service issue where numerical and/oralgorithmic characteristics (such as accuracy, stability androbustness etc.) may need to be added to the performancemodel. Thus the performance of a component implementa-tion would be viewed with respect to the size of the problemas well as the quality of the solution produced by it.

In Figure 6 we plot the communication time spent at dif-ferent levels of the grid hierarchy during each communica-tion (“ghost-cell update”) step. We plot data for processor0 first. During the course of the simulation, the applicationwas load-balanced once, resulting in a different domain de-composition. This is seen in a clustering of message pass-ing times at Level 0 and 2. Ideally, these clusters shouldhave collapsed to a single point; the substantial scatter iscaused by fluctuating network loads. Inset, we plot results

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


Level

Com

mun

icat

ion

Tim

e

0 0.5 1 1.5 2103

104

105

Proc 0

Level

Com

mu

nic

atio

nT

ime

0 0.5 1 1.5 2103

104

105

Proc 0Proc 1Proc 2

Figure 6. Per invocation message passingtime for different levels of the grid hierarchyfor the 3 processors. We see a clustering ofmessage passing times, especially for Levels0 and 2. The grid hierarchy was subjectedto a re-grid step during the simulation whichresulted in a different domain decompositionand consequently the clustering in messagepassing times. Inset : We plot the timingsfor all processors. Similar clustering is ob-served. All times are in microseconds. Sym-bols�, Æ and� indicate data from processors0, 1 and 2.

for all the 3 processors. A similar scatter of data points isseen. Comparing with Figures 4 and 5, we see that mes-sage passing times are generally comparable to the purelycomputational loads of StatesConstructor and EFMFlux,and it is unlikely that the code, in the current configuration(the given problem and the level of accuracy desired) willscale well. This is also borne out by Table 1 where almost aquarter of the time is shown to be spent in message passing.

6 Conclusions

We have proposed a software infrastructure for perfor-mance measurement in HPC component environments. Ourprototypical implementation was used to collect perfor-mance data for a scientific simulation and construct per-formance models. While the data collected is no differ-

ent from what is required in traditional HPC, the measure-ment system must be compatible with component softwaredevelopment methods and new strategies, such as prox-ies, must be adapted from other component-based environ-ments. Proxies can be automatically generated from a com-ponent’s header if the sole purpose is to time the executionof a component. However, for performance modeling, onefrequently needs to record certain inputs to the component.Proxies are the logical place to extract this information be-fore forwarding the component invocation, but this requiresthat this information be identifiable during proxy creation.We are currently investigating simple mark-up approachesidentifying arguments/parameters which affect performanceand need to be extracted and recorded.

The problem of performance modeling is still unsolved.The models derived here are valid only on a similar clus-ter. Any significant change, such as halving of the cachesize, will have a large effect on the coefficients in the mod-els (though the functional form is expected to remain un-changed). Ideally, the coefficients should be parameterizedby processor speed and a cache model. We will address thisin future work, where the cache information collected dur-ing these tests will be employed.

The ultimate aim of performance modeling is to be ableto compose a composite performance model and optimizea component assembly. Apart from performance models,this requires multiple implementations of a functionality (sothat one may have alternates to choose from) and a call tracefrom which the inter-component interaction may be derived.The wiring diagram (available from the framework) alongwith the call trace (detected and recorded by the perfor-mance infrastructure) can be used by the Mastermind tocreate a composite performance model where the variablesare the individual performance models of the componentsthemselves. Figure 7 shows a schematic of how such asystem may construct an abstract dual (represented as a di-rected graph) of the application. Edge weights signify thenumber of invocations and the vertices are weighted by thecompute and communication times, as predicted by the per-formance models of the component implementations. Thecaller-callee relationship is preserved to identify subgraphsthat are insignificant from the performance point of view.This facilitates dynamic performance optimization whichuses online performance monitoring to determine when per-formance expectations are not being met and new model-guided decisions of component use need to take place. Thisis currently underway.

Acknowledgments

This work was supported by the Department of Energy,Office of Science, via the Scientific Discovery through Ad-vance Computing program at Sandia Nationa Loboratories,

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


PM1

PM2

PM3PM4

N12

N13 N34

N43

CCAFFEINE

Mastermind

AbstractFramework

Figure 7. Dual of Figure 1, constructed as adirected graph in the Mastermind, with edgeweights corresponding to the number of invo-cations and the vertex weights being the com-pute and communication times determinedfrom the performance models (PM�) for com-ponent �. Only the port connections shownin black in Figure 1 are represented in thegraph. The Mastermind is seen connected toCCAFFEINE via the AbstractFramework Port toenable dynamic replacement of sub-optimalcomponents.

Livermore. The authors would like to thank the Departmentof Computer Science, University of Oregon, Eugene, forletting us use their “neuronic” cluster for our runs .

Sandia is a multiprogram laboratory operated by SandiaCorporation, a Lockheed Martin Company, for the UnitedStates Department of Energy’s National Nuclear SecurityAdministration under Contract DE-AC04-94AL85000.

References

[1] CORBA Component model webpage. http://www.omg.com.Accessed July 2002.

[2] PAPI: Performance Application Programming Interface.http://icl.cs.utk.edu/projects/papi/.

[3] PCL — The Performance Counter Library. http://www.fz-juelich.de/zam/PCL/.

[4] TAU: Tuning and Analysis Utilities.http://www.cs.uoregon.edu/research/paracomp/tau/.

[5] B. A. Allan, R. C. Armstrong, A. P. Wolfe, J. Ray, D. E.Bernholdt, and J. A. Kohl. The CCA core specifica-tions in a distributed memory SPMD framework. Concur-rency: Practice and Experience, 14:323–345, 2002. Also athttp://www.cca-forum.org/ccafe03a/index.html.

[6] R. Armstrong, D. Gannon, A. Geist, K. Keahey, S. R. Kohn,L. McInnes, S. R. Parker, and B. A. Smolinski. Towarda Common Component Architecture for High-PerformanceScientific Computing. In Proceedings of High PerformanceDistributed Computing Symposium, 1999.

[7] M. J. Berger and P. Collela. Local adaptive mesh refinementfor shock hydrodynamics. J. Comp. Phys., 82:64–84, 1989.

[8] M. J. Berger and J. Oliger. Adaptive mesh refinement forhyperbolic partial differential equations. J. Comp. Phys.,53:484–523, 1984.

[9] R. Englander and M. Loukides. Developing JavaBeans (Java Series). O’Reilly and Associates, 1997.http://www.java.sun.com/products/javabeans.

[10] N. Furmento, A. Mayer, S. McGough, S. Newhouse,T. Field, and J. Darlington. ICENI: Optimisation of Com-ponent Applications within a Grid Environment. ParallelComputing, 28:1753–1772, 2002.

[11] D. J. Kerbyson, H. J. Alme, A. Hoisie, F. Petrini, H. J.Wasserman, and M. L. Gittings. Predictive performanceand scalability modeling of a large-scale application. InProceedings of Supercomputing, 2001. Distributed via CD-ROM.

[12] D. J. Kerbyson, H. J. Wasserman, and A. Hoisie. Explor-ing advanced architectures using performance prediction. InInternational Workshop on Innovative Architectures, pages27–40. IEEE Computer Society Press, 2002.

[13] S. Lefantzi, J. Ray, and H. N. Najm. Using the CommonComponent Architecture to Design High Performance Sci-entific Simulation Codes. In Proceedings of InternationalParallel and Distributed Processing Symposium, 2003.

[14] J. Maloney. Distributed COM Application DevelopmentUsing Visual C++ 6.0. Prentice Hall PTR, 1999. ISBN0130848743.

[15] A. D. Malony and S. Shende. Distributed and Parallel Sys-tems: From Concepts to Applications, chapter PerformanceTechnology for Complex Parallel and Distributed Sys tems,pages 37–46. Kluwer, Norwell, MA, 2000.

[16] A. Mos and J. Murphy. Performance Monitoring of JavaComponent-oriented Distributed Applications. In IEEE 9thInternational Conference on Software, Telecommunicationsand Computer Networks - SoftCOM, 2001.

[17] R. Samtaney and N. Zabusky. Circulation deposition onshock-accelerated planar and curved density stratified inter-faces : Models and scaling laws. J. Fluid Mech., 269:45–85,1994.

[18] S. Shende, A. D. Malony, and R. Ansell-Bell. Instrumen-tation and measurement strategies for flexible and portableempirical performance evaluation. In Proceedings of theInternational Conference on Parallel and Distributed Pro-cessing Techniques and Applications, PDPTA ’2001, pages1150–1156. CSREA, June 2001.

[19] S. Shende, A. D. Malony, C. Rasmussen, and M. Sottile. APerformance Interface for Component-Based Applications.In Proceedings of International Workshop on PerformanceModeling, Evaluation and Optimization, International Par-allel and Distributed Processing Symposium, 2003.

[20] B. Sridharan, B. Dasarathy, and A. Mathur. On Build-ing Non-Intrusive Performance Instrumentation Blocks forCORBA-based Distributed Systems. In 4th IEEE Interna-tional Computer Performance and Depenability Symposium,March 2000.

[21] B. Sridharan, S. Mundkur, and A. Mathur. Non-intrusiveTesting, Monitoring and Control of Distributed CORBA Ob-jects. In TOOLS Europe 2000, June 2000.

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


Figure 8. Snapshot of the component application, as assembled for execution. We see three proxies(for AMRMesh, EFMFlux and States), as well as the TauMeasurement and Mastermind components tomeasure and record performance-related data. Data arrays are allocated in AMRMesh and passedaround among components via pointers.

% Time Exclusive Inclusive # Call Inclusive Namemsec total msec usec/call

100.0 55,244 1:52.032 1 112032939 int main(int, char **)24.3 27,262 27,262 12.75 2138235 MPI Waitsome()12.0 13,482 13,482 1632 8261 g proxy::compute()10.9 12,240 12,240 1632 7501 sc proxy::compute()1.0 1,077 1,077 7029.5 153 icc proxy::prolong()0.8 895 895 186 4813 icc proxy::restrict()0.7 768 768 20959 37 TAU GET FUNCTION VALUES()0.6 662 662 1 662412 MPI Init()0.2 168 168 3.25 51753 MPI Comm dup()0.1 145 145 0.25 581244 MPI Finalize()

Table 1. Top 10 entries from a timing profile done with our infrastructure. Around 50% of the time isaccounted for by g proxy::compute(), sc proxy::compute() and MPI Waitsome(). The MPI callis invoked from AMRMesh. The two other methods are modeled as a part of the work reported here.Timings have been averaged over all the processors. The profile shows the inclusive time (total timespent in the methods and all subsequent method calls), exclusive time (time spent in the specificmethod less the time spent in subsequent instrumented methods), the number of times the methodwas invoked, and the average time per call to the method. % time is calculated from a running sumof the inclusive times.

0-7695-2132-0/04/$17.00 (C) 2004 IEEE


performance measurement and modeling of component

Documents