th e p erformance ev olution of the p ara llel ocean prog ram on … · 2010-08-11 · th e p...

The Performance Evolution of the Parallel Ocean Program on theCray X1 ∗

P. H. Worley †

Oak Ridge National Laboratory

J. Levesque ‡

Cray Inc.

Abstract

We describe our experiences in repeated cycles of performance optimization, benchmarking, andperformance analysis of the Parallel Ocean Program (POP) on the Cray X1 at Oak Ridge NationalLaboratory. We discuss the implementation and performance impact of Co-Array Fortran replacementsfor communication latency-sensitive routines. We also discuss the performance evolution of the systemsoftware from May 2003 to May 2004, and the impact that this had on POP performance.

1 Introduction

The X1 is the first of Cray’s new scalable vec-tor systems [6]. The X1 is characterized by high-speed custom vector processors, high memory band-width, and a high-bandwidth, low-latency intercon-nect linking the nodes. The performance of the pro-cessors in the Cray X1 is comparable or superiorto that of the NEC SX-6 processors in the EarthSimulator on many computational science applica-tions. A significant feature of the Cray X1 is thatit attempts to combine the processor performanceof traditional vector systems with the scalability ofmodern microprocessor-based architectures.

The Parallel Ocean Program (POP) is the oceancomponent of of the Community Climate SystemModel (CCSM) [2], the primary model for globalclimate simulation in the U.S. POP has provenamenable to vectorization on the NEC SX-6, andthe expectation was that the same holds true on theCray X1. This was demonstrated last year by Jones,et al [12] using the SX-6 port of POP on the X1.In this paper we describe in more detail the per-formance impact of the SX-6 inspired vectorizationand MPI [13] optimizations on the Cray X1. We

then describe the performance evolution of the CrayX1 over the past year (May 2003 to May 2004) asadditional optimizations were introduced and as thesystem software on the Cray X1 matured.

2 POP Description

POP is an ocean circulation model derived fromearlier models of Bryan [3], Cox [5], Semtner [1]and Chervin [4] in which depth is used as thevertical coordinate. The model solves the three-dimensional primitive equations for fluid motionson the sphere under hydrostatic and Boussinesq ap-proximations. Spatial derivatives are computed us-ing finite-difference discretizations which are formu-lated to handle any generalized orthogonal grid on asphere, including dipole [17] and tripole [14] gridswhich shift the North Pole singularity into landmasses to avoid time step constraints due to gridconvergence.

Time integration of the model is split intotwo parts. The three-dimensional vertically-varying(baroclinic) tendencies are integrated explicitly us-ing a leapfrog scheme. The very fast vertically-

∗This research was sponsored by the Office of Mathematical, Information, and Computational Sciences, Office of Science,U.S. Department of Energy under Contract No. DE-AC05-00OR22725 with UT-Batelle, LLC. Accordingly, the U.S. Governmentretains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others todo so, for U.S. Government purposes.

†Computer Science and Mathematics Division, Oak Ridge National Laboratory, P.O. Box 2008, Bldg. 5600, Oak Ridge, TN37831-6016 ([email protected])

‡10703 Pickfair Drive Austin, TX 78750 ([email protected])

2 Proceedings of the 46th Cray User Group Conference, May 17-21, 2004

uniform (barotropic) modes are integrated using animplicit free surface formulation in which a precon-ditioned conjugate gradient solver is used to solvefor the two-dimensional surface pressure.

A wide variety of physical parameterizations andother features are available in the model and are de-scribed in detail in a reference manual distributedwith the code. Because POP is a public code,many improvements to its physical parameteriza-tions have resulted from external collaborations withother ocean modeling groups and such developmentis very much a community effort. Detailed descrip-tions of the numerical discretizations and methodsare described in the reference manual and in previ-ous publications [7, 8, 11].

3 Cray X1 Description

The Cray X1 is hierarchical in processor, memory,and network design. The basic building block isthe multi-streaming processor (MSP), which is ca-pable of 12.8 GFlop/sec for 64-bit operations. EachMSP is comprised of four single-streaming proces-sors (SSPs), each with two 32-stage 64-bit floatingpoint vector units and one 2-way super-scalar unit.The SSP uses two clock frequencies, 800 MHz forthe vector units and 400 MHz for the scalar unit.Each SSP is capable of 3.2 GFlop/sec for 64-bit op-erations. The four SSPs share a 2 MB “Ecache”.

The Ecache has sufficient single-stride band-width to saturate the vector units of the MSP. TheEcache is needed because the bandwidth to mainmemory is not enough to saturate the vector unitswithout data reuse - memory bandwidth is roughlyhalf the saturation bandwidth. This design repre-sents a compromise between non-vector-cache sys-tems, like the SX-6, and cache-dependent systems,like the IBM p690, with memory bandwidths anorder of magnitude less than the saturation band-width. The Cray X1’s cache-based design deviatesfrom the full-bandwidth design model only slightly.Each X1 processor is designed to have more single-stride bandwidth than an SX-6 processor; it is thehigher peak performance that creates an imbalance.A relatively small amount of data reuse, which mostmodern scientific applications do exhibit, should en-able a very high percentage of peak performance tobe realized, and worst-case data access should stillprovide double-digit efficiencies.

The X1 compiler’s primary strategies for usingthe eight vector units of a single MSP are paralleliz-ing a (sufficiently long) vectorized loop or paralleliz-ing an unvectorized outer loop. The effective vector

length of the first strategy is 256 elements, like forthe NEC SX-6. The second strategy, which attacksparallelism at a different level, allows a much shortervector length of 64 elements for a vectorized innerloop. Cray also supports the option of treating eachSSP as a separate processor.

Four MSPs and a flat, shared memory of 16 GBform a Cray X1 node. The memory banks of a nodeprovide 200 GByte/sec of bandwidth, enough to sat-urate the paths to the local MSPs and service re-quests from remote MSPs. Each bank of sharedmemory is connected to a number of banks on re-mote nodes, with an aggregate bandwidth of roughly50 GByte/sec between nodes. This represents a byteper flop of interconnect bandwidth per computationrate, compared to 0.25 bytes per flop on the EarthSimulator and less than 0.1 bytes per flop expectedon an IBM p690 with the maximum number of Fed-eration connections. The collected nodes of an X1have a single system image.

A single four-processor X1 node behaves like atraditional shared memory processor (SMP) node,but each processor has the additional capability ofdirectly addressing memory on any other node (likethe T3E). Remote memory accesses go directly overthe X1 interconnect to the requesting processor, by-passing the local cache. This mechanism is morescalable than traditional shared memory, but it isnot appropriate for shared-memory programmingmodels, like OpenMP [16], outside of a given four-processor node. This remote memory access mech-anism is a good match for distributed-memory pro-gramming models, particularly those using one-sidedput/get operations.

In large configurations, the Cray X1 nodes areconnected in an enhanced 3D torus. This topol-ogy has relatively low bisection bandwidth com-pared to crossbar-style interconnects, such as thoseon the NEC SX-6 and IBM SP. Whereas bisectionbandwidth scales as the number of nodes, O(n), forcrossbar-style interconnects, it scales as the 2/3 rootof the number of nodes, O(n2/3), for a 3D torus.Despite this theoretical limitation, mesh-based sys-tems, such as the Intel Paragon, the Cray T3E, andASCI Red, have scaled well to thousands of proces-sors.

4 Experiment Details

In these experiments we used version 1.4.3of POP as downloaded from the POP coderepository at Los Alamos National Laboratory(http://climate.lanl.gov/Models/POP/index.htm).

Performance Evolution of POP 3

We used a benchmark configuration (called x1) rep-resenting a relatively coarse resolution similar tothat currently used in coupled climate models. Thehorizontal resolution is roughly one degree (320x384)and uses a displaced-pole grid with the pole of thegrid shifted into Greenland and enhanced resolutionin the equatorial regions. The vertical coordinateuses 40 vertical levels with a smaller grid spac-ing near the surface to better resolve the surfacemixed layer. Because this configuration does not re-solve eddies, it requires the use of computationally-intensive subgrid parameterizations. This configu-ration is set up to be identical to the actual pro-duction configuration of the Community ClimateSystem Model with the exception that the couplingto full atmosphere, ice and land models has beenreplaced by analytic surface forcing.

The parallel algorithms in POP are based on adecomposition of the horizontal grid onto a two di-mensional virtual processor grid. Thus, for exam-ple, when using 8 processors, POP can be run ona 1x8, 2x4, 4x2, or 8x1 grid. The choice of proces-sor grid affects both load balance and comunicationoverhead. In all results presented in the paper, theoptimal processor grid was first identified for eachprocessor count, and only these results were used.In consequence, the processor grids used for a givenprocessor count may vary between platforms.

The performance of POP is primarily determinedby the performance of the baroclininc and barotropicprocesses described earlier. The baroclinic processis three dimensional with limited nearest-neighborcommunication and typically scales well on all plat-forms. In contrast, the barotropic process involves atwo-dimensional, implicit solver whose performanceis very sensitive to network latency and typicallyscales poorly on all platforms.

As described later, version 1.4.3 was ported tothe Earth Simulator by targeted vectorization andoptimization of MPI communications. For the X1,we based our initial optimizations on this Earth Sim-ulator version. Further optimizations came primar-ily from the introduction of Co-Array Fortran [15]implementations of the latency-sensitive communi-cations in the barotropic process.

POP is instrumented with calls to MPI WTIMEto record the time spent in the basic logical units,such as the baroclinic and barotropic processes. Formore detailed performance analyses we used theMPICL profiling and tracing package [9, 19], theParagraph visualization tool [10], and the Cray Per-formance Analysis Tool (PAT).

5 Platforms

POP performance results are presented for a CrayX1, the Earth Simulator, an HP AlphaServer SC, anIBM p690 cluster, an IBM SP, and an SGI Altix withthe following processor and system specifications.

Processor Proc Cache Memory/Peak Proc

(GF/s) (MB) (MB)Cray X1Cray 12.8 2 4000Earth SimulatorES 8.0 N/A 2000HP AlphaServer SCEV68 2.0 8 (L2) 1000IBM p690 clusterPower4 5.2 1.5 (L2) 1000IBM SPPower3-II 1.5 8 (L2) 1000SGI AltixItanium2 6.0 6 (L3) 8000

SMP Size Switch/NetworkCray X14 CrayEarth Simulator8 ESHP AlphaServer SC4 Quadrics QsNetIBM p690 cluster32 SP Switch2 (Corsair)IBM SP16 SP Switch2 (Colony)SGI Altix256 NUMAlink

The IBM p690 cluster, Cray X1, and the SGI Al-tix are located in the Center for Computational Sci-ences at Oak Ridge National Laboratory (ORNL).The HP AlphaServer SC is located at the Pitts-burgh Supercomputer Center (PSC). The IBM SPis located in the National Energy Research ScientificComputing Center (NERSC) at Lawrence BerkeleyNational Laboratory. The Earth Simulator is housedin the Earth Simulator Center in Yokohama, Japan.

The Cray X1 at ORNL has grown from the ini-tial 32 processor (MSP) system delivered in March2003, to a 128 MSP system in July 2003, to a 256MSP system in October 2003, and to a 512 MSP sys-tem in June 2004. Results are presented throughoutthis period, ending with the 256 MSP system in May2004.


6 Vectorization

Figure 6.1 compares the performance of POP forthe fixed size x1 benchmark on a number of differ-ent platforms. The graphs are plots of simulationyears per day of computation as a function of thenumber of processors (MSPs for the Cray X1). TheEarth Simulator results are a version of POP thatwas ported and optimized on the Earth Simulatorby Dr. Y. Yoshida of the Central Research Instituteof the Electric Power Industry (CRIEPI). The Altixresults are for a version of POP that uses the sameMPI optimizations used on the Earth Simulator. Allother platforms used the stock POP version 1.4.3.(The MPI optimizations used on the Altix do notchange the performance on the IBM or HP systemssignficantly.) The performance of the stock versionof POP is approximately the same on the X1 as onthe HP and IBM systems, and much worse than onthe Earth Simulator or the SGI Altix.

10

20

30

40

50

60

70

80

90

100

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors

LANL Parallel Ocean Program POP 1.4.3, x1 benchmark

Earth Simulator [vector version]SGI Altix (1.5 GHz)IBM p690 cluster (1.3 GHz)HP AlphaServer SC (1.0 GHz)Cray X1 [original version]IBM SP (375 MHz)

Figure 6.1: POP platform comparison: initialresults

The first step we took in optimizing POP forthe Cray X1 was to exploit the vectorization workalready done by Yoshida. For the Earth Simulatorport, Yoshida modified 701 lines of POP source code(out of 45000 lines) to improve vectorization. Overhalf of these modifications (approx. 400 lines) in-volved replacing Fortran 90 where, merge, shift,etc. constructs with their Fortran 77 equivalents.While these latter changes do not improve (or de-grade) performance on the Cray X1, we adoptedthem anyway to minimize the differences betweenthe two vector versions of the code. We then ex-amined and changed some of the NEC compiler di-rectives to Cray compiler directives. We also modi-fied two routines (that were previously modified forthe Earth Simulator) to improve performance on theCray. Changes in one routine involved a few sim-

ple loop reorderings, to enable the Cray to streamover outer loops. Changes in the other routine in-volved undoing some of the promotion of scalar tem-poraries to vector temporaries, resulting in code thatwas closer to the unvectorized version of POP.

Figure 6.2 contains task Gannt charts for POPwhen using 128 MSPs on the X1. The top chartdescribes performance for the unmodified version ofPOP 1.4.3, while the bottom chart describes perfor-mance after making the modifications for vectoriza-tion. The X-axis is time, where each unit represents2 milliseconds. The Y-axis is the process (MSP) id.The chart describes the task that each process is ex-ecuting at a given point in time. The tasks are asfollows:

Task Task Description0 everything else

(primarily tracer updates)1 baroclinc2 baroclinic boundary update3 barotropic (excluding the solver)4 barotropic boundary update5 barotropic solver

From these data, vectorization primarily decreasedthe time spent in the baroclinic process, approxi-mately tripling the performance of this phase of thecomputation.

Figure 6.3 contains utilization count graphs forthe same period of execution time for the original(top) and vectorized (bottom) versions of POP. TheY-axis is the number of processes in one of threestates: busy (computing), overhead (actively com-municating in an MPI command), idle (blockedwaiting to receive a message). While subject tosome error, overhead + idle does accurately re-flect the time spent in MPI commands. From thesedata, most of the time in a 128 MSP run is spentin interprocessor communication. Also, vectoriza-tion improved performance of the computation inthe baroclinic process by approximately a factor of10. The performance of the baroclinic process as awhole increased by a factor of only three because thebaroclinic communication overhead was unchanged.


Figure 6.2: Task graph: without and withvectorization

Figure 6.3: Utilization count: without and withvectorization

7 MPI Optimizations

It is clear from Fig. 6.3 that communication costsdominate in the 128 MSP performance data. Thenext step in the Cray X1 optimization was to min-imize these costs. While there was no reason toexpect that the Earth Simulator-specific MPI op-timizations to be useful on the X1, it was an easyexperiment to make. In fact, these modificationsimproved performance on the Cray X1 significantly.

For the Earth Simulator, Dr. Yoshida modified125 lines and added one new (140 line) routine to im-prove MPI performance. In the original version ofPOP, three routines in the barotropic process usedMPI derived datatypes when updating halo regionsin the domain decomposed data structures. Thislogic was replaced with explicitly packing and un-packing contiguous communication buffers and usingstandard datatypes in MPI calls. In the baroclinicprocess and in the baroclinic correct adjust

routine, which updates tracers, a loop over a se-ries of halo updates was replaced by a single rou-tine that combined these into fewer communicationcalls involving larger messages. Finally, five routineswere modified to replace communication logic usingMPI IRECV and MPI ISEND with similar logic usingMPI ISEND and MPI RECV. This latter modificationdoes not improve (or degrade) performance on theX1, but was again adopted in order to minimize dif-ferences with the Earth Simulator version of POP.

Figure 7.1 contains the task Gannt chart for thevectorized version, without (top) and with (bottom)these MPI optimizations. The X-axis resolution istwice that used in in Figs. 6.2 and 6.3, with 1 mil-lisecond units. Figure 7.2 contains the correspond-ing utilization count graphs. From these data, elim-inating derived types triples the performance of thebarotropic solver. Combining the halo updates al-most eliminates the time spent doing tracer updatesand more than halves the time spent in the baro-


clinic process. From the utilization count graphs,POP on the X1 is still communication bound whenusing 128 MSPs for this benchmark problem, butthe performance is much improved.

Figure 7.3 demonstrates the performance impactof vectorization and MPI optimization on the X1.Using this modified version of the Earth Simulatorport, performance on the X1 is similar to that on theEarth Simulator up to 96 processes. This version ofthe code is very similar to that used in the earlierpaper [12]. Performance reported here is superiordue to OS and other system software improvementsover the past year (April 2003 to May 2004). Thiswill be discussed in more detail in later sections.

Figure 7.1: Task graph: without and with MPIoptimization

Figure 7.2: Utilization count: without and withMPI optimization

10

20

30

40

50

60

70

80

90

100

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors


Earth Simulator (vector version)Cray X1 (vector version)SGI Altix (1.5 GHz)IBM p690 cluster (1.3 GHz)HP AlphaServer SC (1.0 GHz)IBM SP (375 MHz)

Figure 7.3: Platform comparison: withvectorization and MPI optimization


8 Co-Array Fortran Optimiza-tions

Process scaling for a fixed size problem will necessar-ily show diminishing returns for large process counts.However, the feeling was that the rollover in X1 per-formance in Fig. 7.3 occurred too soon. Performancebefore the rollover is also lower than expected com-pared to the Earth Simulator given the peak proces-sor and network rates on the Cray X1 and the EarthSimulator. From Fig. 7.2, communication overheadis the dominant constraint on performance. Profilingindicates that dominant communication overhead inthis port of POP to the X1 is from

• GLOBAL SUM: global sum of the “physical do-main” of a two dimensional array. This is usedto compute the inner product in the conjugategradient solver in the barotropic process. TheMPI version uses MPI Allreduce.

• NINEPT 4: weighted nearest neighbor sum for9 point stencil, requiring a halo update. Thisis used to compute residuals in the conjugategradient solver in the barotropic process.

The performance of both of these is sensitive to MPIlatency. MPI performance for small messages is aknown problem on the X1, and the current solutionis to use Co-Array Fortran to implement latency-sensitive communications. Co-Array Fortran moreefficiently exploits the globally addressable memoryon the Cray than is currently possible with MPI two-sided messaging semantics. For example, Fig. 8.1is a graph of performance of a halo update (usingWallcraft’s HALO benchmark [18]) for 16 MSPs asa function the number of four byte words in the halo.For small halos, the Co-Array Fortran implementa-tion is over 5 times faster.

MPI and Co-Array Fortran messaging can co-exist in the same program. For POP we replaced theexisting inner product and halo update subroutinesused in barotropic solver with Co-Array Fortran im-plementations. While performance improved withour initial implementations, it was not as good aswe expected. Over the next 8 months (May 2004 toDecember 2004) we developed and tested a numberof different implementations:

10

100

1000

10000

1 10 100 1000 10000 100000 1e+06

Mic

rose

cond

s

Words

HALO Performance on 16 MSPs of the Cray X1

MPISHMEM

Co-Array Fortran

Figure 8.1: Paradigm Comparison for HaloUpdate

May 7

• GLOBAL SUM: Master reads partial sums fromremote memory, completes sum, and writes re-sults back to other processes’ memory. Globalbarriers are used for synchronization.

• NINEPT 4: Each process reads remote memoryto update the local halo. East-west updatesoccur first, followed by north-south updates.Global barriers are used for synchronization.

August 12

• GLOBAL SUM: Processes write partial sums tomaster’s memory. Flags are used to implementpairwise synchronization, eliminating the needfor global barriers.

• NINEPT 4: Each process writes remote mem-ory to fill remote halos. East-west updatesoccur first, followed by north-south updates.Flags are used to implement pairwise synchro-nization, eliminating the need for global barri-ers.

September 1

• GLOBAL SUM: Added padding to co-arrays inAugust 12 version to eliminate contention inthe memory controller.

• NINEPT 4: Added padding to co-arrays in Au-gust 12 version to eliminate contention in thememory controller.


Figure 8.2: Task graph: without and withCo-Array Fortran optimization

Figure 8.3: Utilization count: without and withCo-Array Fortran optimizations

October 25

• GLOBAL SUM: Used special values in data co-arrays to implement pairwise synchronizationin September 1 version, eliminating flags.

• NINEPT 4: Used September 1 algorithm.

December 13a

• GLOBAL SUM: Used a variant of the May 7 ver-sion in which partial sums are written to themaster’s memory. Global barriers are againused for synchronization.

• NINEPT 4: Used August 12 algorithm.

December 13b

• GLOBAL SUM: Used a generalization of theSeptember 1 algorithm that implements a treesum and tree broadcast. The branching factor

is specified at compile time. For the resultsdescribed here, each parent has up to 16 chil-dren.

• NINEPT 4: Used August 12 algorithm.

Figures 8.2-8.4 compare the performance of theoptimized MPI-only version with the optimized ver-sion using the December 13b Co-Array Fortran mod-ifications. Figures 8.2 and 8.3 are the usual taskGannt chart and utilization count graphs. The X-axis resolution is again doubled from the previousfigures, with .5 millisecond units. The Co-Array For-tran modifications were used only in the barotropicsolver, and this is the only task that shows any per-formance change. However, when using 128 MSPs,the barotropic solver is 5 times faster when usingthe Co-Array Fortran modifications, and is no longercommunication bound. (In Fig. 8.3, the time spentin GLOBAL SUM and NINEPT 4 is defined to be idle,as this is closer to reality than defining it to be


busy. Fine grain tracing of the Co-Array Fortranwould perturb performance too much to be use-ful.) Figure 8.4 is task Gannt chart focusing on thebarotropic solver. The X-axis unit is 1 microsecond.The three tasks are 1: GLOBAL SUM, 2:NINEPT 4, and0: everything else. From this, the Co-Array Fortranimplementation is 8 times faster than the MPI im-plementation for GLOBAL SUM and 4 times faster forNINEPT 4.

Figure 8.4: Task graph of barotropic solver:without and with Co-Array Fortran optimization

9 System Software Optimiza-tions

As mentioned earlier, the development and evalua-tion of the Co-Array Fortran implementations tookplace over 8 months. Figure 9.1 is a graph of theperformance evolution that we have observed overthe past year using the best performing version ofPOP at each point in time. While the performanceof POP motivated the algorithm experimentation,

some of the performance improvement did not arisefrom algorithm improvements. This can be seenin the performance improvement in the December13b version between March 4 and May 11, 2004.The source of each non-algorithmic performance im-provement varies with the particular update to thesystem software, but one of the largest changes wasmade in a September 1 update to the operating sys-tem. This is displayed in Fig. 9.2. On August 29,the August 12 version of POP achieved a maximumperformance of 85 years per day, but the average per-formance was much lower. The variability of POPtimings was very high. This behavior was used todocument a problem in the way that the operatingsystem scheduled interrupts. By using a global clockto schedule interrupts on all processors, the Septem-ber 29 performance was achieved. The best perfor-mance increased from 85 years per day to 125 yearsper day, and the performance variability decreasedsignificantly.

20

40

60

80

100

120

140

160

180

200

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors

LANL Parallel Ocean Program POP 1.4.3, x1 benchmark, on the Cray X1

May 11, 2004 Vec. POP (Dec13b version)Mar. 4, 2004 Vec. POP (Dec13b version)Oct. 19, 2003 Vec. POP (Aug12 version)Sep. 25, 2003 Vec. POP (Sep1 version)Aug. 12, 2003 Vec. POP (Aug12 version)July 23, 2003 Vec. POP (May7 version)May 7, 2003 Vectorized POP with tuned MPI Unmodified POP 1.4.3

Figure 9.1: Performance evolution

20

40

60

80

100

120

140

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors

LANL Parallel Ocean Program POP 1.4.3, x1 benchmark on the Cray X1

Vec. POP (Aug12 version) Sep. 29 timings best timings average timings Aug. 29 timings best timings average timings

Figure 9.2: Performance Impact of Sept. 1 OSUpdate


Figure 9.3 is a graph of the performance of theAugust 12 implementation up to May 11, 2004. Fig-ure 9.4 is a graph of the performance of the opti-mized MPI-only implementation for the same pe-riod. In both cases, it is apparent that the perfor-mance of the operating system and system softwarehas continued to improve. In particular, while theMPI-only version is still not competitive with theversion using Co-Array Fortran optimizations, MPIperformance has improved significantly.

20

40

60

80

100

120

140

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors


Vec. POP (Aug12 version) May 11, 2004 Oct. 19, 2003 Sep. 29, 2003 Aug. 29, 2003

Figure 9.3: Performance Impact of OS Updateson Aug. 12 version

10

20

30

40

50

60

70

80

90

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors


Vec. POP (MPI-only version) May 11, 2004 Mar. 15, 2004 Oct. 19, 2003 Sep. 25, 2003 May 7, 2003

Figure 9.4: Performance Impact of OS Updateson MPI-only version

Figure 9.5 is a graph of the performance of allof the POP implementation described so far as ofMay 11, 2004. Notice that the Co-Array Fortranoptimizations prior to December 13 have very sim-ilar performance. The attempt to solve what wereultimately performance bottlenecks in the operat-ing system by algorithm optimization was not veryeffective. It was not until a number of operating sys-tem problems were resolved that significant progress

could be made. For example, a tree-based algorithmsimilar to that used in the December 13b version wastried in September. It was not competitive at thattime, and was discarded.

20

40

60

80

100

120

140

160

180

200

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors


May 11, 2004 timings Vec. POP (Dec13b version) Vec. POP (Dec13a version) Vec. POP (Oct25 version) Vec. POP (Sep1 version) Vec. POP (Aug12 version) Vec. POP (May7 version) Vec. POP (MPI only) Unmodified POP 1.4.3

Figure 9.5: Implementation Comparison

Figure 9.6 is a graph of the performance of justthe barotropic process for all of the POP implemen-tations (in terms of wallclock seconds per simulationday). The advantage of the December 13 versionsof POP is their scalability. The time spent in thebarotropic is beginning to increase for large processcounts for all of the other versions of POP. For theDecember 13 versions, the time is relatively constantfor 64 to 240 processes. Figure 9.7 is a graph of thewallclock seconds per simulation day for both thebarotropic and baroclinic processes, for both the op-timized MPI-only and December 13b versions. Theperformance (and code) for the baroclinc process isidentical for these two implementations, and is scal-ing well out to 240 processes. For the MPI-only ver-sion, the time spent in the barotropic process startsgrowing at 16 processes and dominates that of thebaroclinc when using more than 92 processes. Incontrast, for the December 13b version, time spent inbarotropic process stops decreasing at 92 processes,but has not yet started growing at 240 processes. Itis also only one third the time spent in the baro-clinic process when using 240 processes. From thisdata, performance should continue to scale to 256processes and beyond.


0.25

0.5

1

2

4

8

16

32

1 2 4 8 16 32 64 128 256

Seco

nds

per S

imul

atio

n Da

y

Processors

POP Barotropic Timings on the Cray X1

May 11, 2004 timings Unmodified POP 1.4.3 Vec. POP (MPI only) Vec. POP (May7 version) Vec. POP (Aug12 version) Vec. POP (Sep1 version) Vec. POP (Oct25 version) Vec. POP (Dec13a version) Vec. POP (Dec13b version)

Figure 9.6: Implementation Comparison:Barotropic Process

0.25

0.5

1

2

4

8

16

32

64

128

1 2 4 8 16 32 64 128 256 512

Seco

nds

per S

imul

atio

n Da

y

Processors

POP Baroclinic and Barotropic Timings

Vector Version (MPI) Baroclinic BarotropicVector Version (MPI and Co-Array Fortran) Baroclinic Barotropic

Figure 9.7: Implementation Comparison:MPI-only vs. Dec13b

10 Platform Comparisons

Figure 10.1 is the current platform comparison us-ing the x1 benchmark problem and version 1.4.3 ofPOP. It includes performance data for the originalversion, the optimized MPI-only version, and theDecember 13b version of POP on the Cray X1. Per-formance on the X1 is clearly better than that onthe IBM and HP systems. In particular, no mat-ter how many processors are used on the IBM andHP systems (for this fixed size benchmark), perfor-mance will never approach that on the X1. Onlyperformance on the Earth Simulator and the SGIAltix among these platforms approach the perfor-mance on the X1.

20

40

60

80

100

120

140

160

180

200

1 2 4 8 16 32 64 128 256

Sim

ulat

ion

Year

s pe

r Day

Processors


Cray X1 (MPI and Co-Array Fortran)Earth SimulatorCray X1 (MPI-only)SGI Altix (1.5 GHz)IBM p690 cluster (1.3 GHz)HP AlphaServer SC (1.0 GHz)Cray X1 (orig. version)IBM SP (375 MHz)

Figure 10.1: Platform comparison: currentresults

Figure 10.2 contains graphs of the performanceof the barotropic and baroclinic processes on the X1and the Earth Simulator. From these data, EarthSimulator performance is better for the baroclinic forthe smallest process counts, and worse for the largestprocess counts. This (roughly) indicates a perfor-mance advantage on the Earth Simulator for longvectors and an advantage on the X1 for short vec-tors. As the vectorization in POP is basically thatdeveloped for the Earth Simulator, this may alsoindicate that more Cray-specific optimizations areneeded. For the barotropic process, the X1 perfor-mance is superior when using more than 8 processes.The Earth Simulator has 8-processor SMP nodes,and messaging between SMP nodes is required whenusing more than 8 processors. The performance ofthe barotropic process on the Earth Simulator is rel-atively constant out to 128 processes, but it also be-gins dominating the baroclinc process at this point.Using more than 128 processors on the Earth Sim-ulator is unlikely to improve performance for thisbenchmark problem. Note that the Earth Simulatorperformance data are from March, 2003. The EarthSimulator implementation could also have continuedto evolve, perhaps by using MPI-2 1-sided messag-ing in the barotropic solver, but we are not aware ofmore recent performance data.


0.25

0.5

1

2

4

8

16

32

64

128

1 2 4 8 16 32 64 128 256 512

Seco

nds

per S

imul

atio

n Da

y

Processors


Earth Simulator Baroclinic BarotropicCray X1 Baroclinic Barotropic

Figure 10.2: Platform comparison: X1 vs. ES

Figure 10.3 contains graphs of the performancethe barotropic and baroclinic processes on the X1and the SGI Altix. The X1 is approximately 5 timesfaster than the Altix for the baroclinic process whenusing 8 processors, and 3 times faster when using 240processors. The shorter vector lengths are decreas-ing the performance advantage of the vector pro-cessor over that of the Itanium2. The X1 is alsoapproximately 3 times faster than the Altix for thebarotropic process when using 240 processors. Timespent in the barotropic is relatively flat for 64 to 240procssors. So, while both systems will likely be ableto use more processors effectively, the SGI does notappear to be able to approach closer than one thirdof the performance of the X1. Note that POP ver-sion 1.4.3 comes with an implementation that usesSHMEM for interprocessor communication. Thisversion does not improve POP performance on theSGI. Other optimizations may also be possible, butwe are already using the Yoshida MPI optimizationson the Altix.

0.25

0.5

1

2

4

8

16

32

64

128

1 2 4 8 16 32 64 128 256 512

Seco

nds

per S

imul

atio

n Da

y

Processors


SGI Altix (1.5 GHz) Baroclinic BarotropicCray X1 Baroclinic Barotropic

Figure 10.3: Platform comparison: X1 vs. Altix

11 Conclusions

We learned a number of important lessons from op-timizing and tracking POP performance on the CrayX1. First, vector and MPI optimizations developedfor the Earth Simulator were reasonably efficient onthe X1, and were a good starting point when opti-mizing for the Cray. While some of these optimiza-tions did not enhance performance, they also didnot degrade performance. However, the advantageof the Earth Simulator for long vector performanceindicates that we need to revisit Cray-specific vector-ization, especially for larger problem sizes. Second,explicit packing and unpacking of message buffersand using standard data types performs much betterthan using MPI derived datatypes to do this implic-itly (currently). Third, the performance of latency-sensitive MPI collective and point-to-point com-mands limits scalability of latency-sensitive codes(again, currently). Co-Array Fortran can be used towork around this deficiency in MPI performance andachieve excellent scalability. Fourth, performanceaspects of the operating system and other systemsoftware improved significantly from May 2003 toMay 2004. Finally, we found tracking the perfor-mance evolution of POP to be an effective way ofidentifying performance problems in the operatingsystem and other system software.

Future work includes optimization and bench-marking for even higher processor counts and forlarger benchmark problems. In particular, POP willbe used for additional investigations into Co-ArrayFortran algorithms and performance, operating sys-tem scalability, and MPI performance as the systemsoftware continues to evolve.

12 Acknowledgements

We gratefully acknowledge

• Dr. Phil Jones of Los Alamos National Labo-ratory for help in obtaining and porting POPand for defining benchmark problems;

• Dr. Tushar Mohan of Lawrence Berkeley Na-tional Laboratory for providing performancedata from the IBM SP at NERSC;

• Howard Pritchard and James Schwarzmeier ofCray Inc. and James B. White III of ORNL fortheir contributions to the development of theCo-Array Fortran algorithms used in POP;


• Dr. Yoshikatsu Yoshida of CRIEPI for his ex-cellent port of POP to the Earth Simulatorand for providing the performance data fromthe Earth Simulator.

We also thank the Pittsburgh Supercomputer Cen-ter for access to the HP AlphaServer SC and theORNL Center for Computational Science for accessto the Cray X1, IBM p690 cluster, and SGI Altix.

13 About the Authors

John Levesque is a 30 year veteran of High Per-formance Computing. Levesque is Technical Staffto the VP of Cray Development and is responsi-ble for demonstrating performance capabilities ofCray’s HPC systems. In addition to porting and op-timizing applications for Cray platforms, Levesquegives training workshops on the effective utilizationof Cray Hardware. He can be reached at the USmail address listed on the title page or via E-mail:[email protected].

Pat Worley is a Senior Research Scientist in theComputer Science and Mathematics Division at OakRidge National Laboratory. He has been conduct-ing early evaluations of advanced computer archi-tectures since the early 90’s. He also does researchin performance evaluation tools and methodologies,and designs and implements parallel algorithms inclimate and weather models. He can be reached atthe US mail address listed on the title page or viaE-mail: [email protected].

References

[1] J. A. Semtner Jr., Finite-difference formula-tion of a world ocean model, in Advanced Phys-ical Oceanographic Numerical Modeling, J. J.O’Brien, ed., Reidel Publishing Company, Nor-well, MA, 1986, pp. 187–202. NATO ASI Series.

[2] M. B. Blackmon, B. Boville, F. Bryan,R. Dickinson, P. Gent, J. Kiehl,R. Moritz, D. Randall, J. Shukla,S. Solomon, G. Bonan, S. Doney,I. Fung, J. Hack, E. Hunke, and J. Hur-rel, The Community Climate System Model,BAMS, 82 (2001), pp. 2357–2376.

[3] K. Bryan, A numerical method for the studyof the circulation of the world ocean, J. Comp.Phys., 4 (1969), pp. 347–376.

[4] R. M. Chervin and J. A. Semtner Jr.,An ocean modeling system for supercomputerarchitectures of the 1990s, in Proceedings ofthe NATO Advanced Research Workshop onClimate-Ocean Interaction, M. Schlesinger, ed.,Kluwer Academic Publishers, Dordrecht, TheNetherlands, 1988, pp. 565–568.

[5] M. D. Cox, A primitive equation, 3-dimensional model of the ocean, Tech. Rep.GFDL Ocean Group Technical Report No. 1,GFDL/NOAA, Princeton, NJ, 1984.

[6] Cray Inc., Cray X1.http://www.cray.com/products/systems/x1/.

[7] D. K. Dukowicz and R. D. Smith, Im-plicit free-surface method for the bryan-cox-semtner ocean model, J. Geophys. Res., 99(1994), pp. 7991–8014.

[8] D. K. Dukowicz, R. D. Smith, and R. C.Malone, A reformulation and implementationof the bryan-cox-semtner ocean model on theconnection machine, J. Atmos. Ocean. Tech.,10 (1993), pp. 195–208.

[9] G. A. Geist, M. T. Heath, B. W. Pey-ton, and P. H. Worley, A users’ guideto PICL: a portable instrumented communica-tion library, Tech. Rep. ORNL/TM-11616, OakRidge National Laboratory, Oak Ridge, TN,August 1990.

[10] M. T. Heath and J. A. Etheridge, Visu-alizing performance of parallel programs, Tech.Rep. ORNL/TM-11813, Oak Ridge NationalLaboratory, Oak Ridge, TN, May 1991.

[11] P. W. Jones, The Los Alamos Parallel OceanProgram (POP) and coupled model on MPPand clustered SMP computers, in Making itsMark – The Use of Parallel Processors in Mete-orology: Proceedings of the Seventh ECMWFWorkshop on Use of Parallel Processors in Me-teorology, G.-R. Hoffman and N. Kreitz, eds.,World Scientific Publishing Co. Pte. Ltd., Sin-gapore, 1999.

[12] P. W. Jones, P. H. Worley, Y. Yoshida,J. B. W. III, and J. Levesque, Practical per-formance portability in the Parallel Ocean Pro-gram (POP), Concurrency and Computation:Experience and Practice, ((in press)).

[13] MPI Committee, MPI: a message-passing in-terface standard, Internat. J. SupercomputerApplications, 8 (1994), pp. 165–416.


[14] R. J. Murray, Explicit generation of orthogo-nal grids for ocean models, J. Comp. Phys., 126(1996), pp. 251–273.

[15] R. W. Numrich and J. K. Reid, Co-ArrayFortran for parallel programming, ACM FortranForum, 17 (1998), pp. 1–31.

[16] OpenMP Architecture Review Board,OpenMP: A proposed standard api forshared memory programming. (availablefrom http://www.openmp.org/openmp/mp-documents/paper/paper.ps), October 1997.

[17] R. D. Smith, S. Kortas, and B. Meltz,Curvilinear coordinates for global ocean mod-els, Tech. Rep. LAUR-95-1146, Los Alamos Na-tional Laboratory, Los Alamos, NM, 1995.

[18] A. J. Wallcraft, SPMD OpenMP vs MPIfor Ocean Models, in Proceedings of theFirst European Workshop on OpenMP,Lund, Sweden, 1999, Lund University.http://www.it.lth.se/ewomp99.

[19] P. H. Worley, MPICL.http://www.csm.ornl.gov/picl/.

th e p erformance ev olution of the p ara llel ocean prog ram on … · 2010-08-11 · th e p...

Documents