oasis3 and oasis4 : the prism couplers g. berti, p. bourcier, a. caubel, d. declat, m.-a. foujols,...

FLUME-PRISM Workshop Met Office, Exeter, March 15-16, 2004

OASIS3 and OASIS4 : the PRISM couplers

G. Berti, P. Bourcier, A. Caubel, D. Declat, M.-A. Foujols, J. Latour, S. Legutke,

J. Polcher, R. Redler, H. Ritzdorf, T. Schoenemeyer, S. Valcke and R. Vogelsang

FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004 Slide 2

Outline

OASIS historicOASIS community todayPRISM first coupler: Oasis3

•model adaptation•coupled model configuration•communication•interpolations/transformations

PRISM final coupler: Oasis4•model adaptation•the Driver•communication•interpolations/transformations•in the prototype•still to be done

Conclusions


OASIS: Ocean Atmosphere Sea Ice Soil

OASIS historic

Developed since 1991 in CERFACS to couple existing GCMs. At the time:

•Models at relatively low resolution (~10000-20000 pts)

•Small number of 2D coupling fields (~10) •Low coupling frequency (~once/day)

flexibility was very important, efficiency not so much!


1993: OASIS 1.0 (CRAY pipes & files)

1995: OASIS 2.0 (PVM3): Heterogeneous coupling (Cathode)

1996: OASIS 2.1: Global warning scenario (SIDDACLICH)

1998: OASIS 2.2 (SIPC): Seasonal forecasting (Demeter)

1999: OASIS 2.3 (GMEM - NEC): Intercomparison (SINTEX)

2000: OASIS 2.4 (MPI2): Climate Natural variability (Predicate)

2003: OASIS 3 (MPI2-MPI1 + PSMILe + I/O): PRISM

=> new Oasis3 (oasis3_prism_2-1) now available !

OASIS historic


•CERFACS (France) ARPEGE3 - ORCA2LIM ARPEGE3 - OPA 8.1 ARPEGE3 - OPAICE

•METEO-FRANCE (France)ARPEGE4 - ORCA2 ARPEGE medias -OPAmed ARPEGE3 - OPA8.1 ARPEGE2 - OPA TDH

•IPSL- LODYC, LMD, LSCE (France) LMDz - ORCA2LIMLMDz - ORCA4 LMDz - OPA ATL3/ATL1 IFS

- OPA 8.1 ECHAM4 - ORCA2

•MERCATOR (France) PAM(OPA)

•MPI - M&D (Germany)ECHAM5 - MPI-OM ECHAM5 - C-HOPE PUMA - C-HOPE EMAD - E-HOPE ECHAM5 - E-HOPE

ECHAM4 - E-HOPE

•ECMWF (UK)IFS Cy23r4 - E-HOPE IFS Cy15r8 - E-HOPE

OASIS community today


•CGAM-Reading (UK) HadAM3 - ORCA2

•Southampton University (UK) Inter. Atm - OCCAM lite

•UCL (Belgium) LMDz - CLIO

•SMHI (Sweden) ECHAM - RCA RCA(region.) – RCO(region.)

•U. of Bergen (Norway) MM5 - ROMS

•KNMI (Netherlands) ECHAM5 - MPI-OM

•DMI (Danemark) ECHAM - HIRLAM

•INGV (Italy) ECHAM5 – MPI-OM

•IRI (USA) ECHAM4 - MOM3

•JAMSTEC (Japan) ECHAM4 - OPA 8.2

•BMRC (Australia) BAM - MOM4 BAM3 - ACOM2

•U. of Tasmania (Australia) Data Atm. - MOM4

•CAS,IIT Delhi (India) MM5 - POM

OASIS community today


PRISM first coupler: Oasis3

Mono-process coupler + parallel coupling library (PSMILe)

• synchronisation of the component models • coupling fields exchange • I/O actions• mono-process interpolation

Flexibility, modularity:• Arbitrary number of models and coupling fields • Different coupling frequencies for each field• Different transformations for each field

Static coupling: all coupling parameters defined initially.


Oasis3: model adaptation (1/3)

PRISM System Model Interface Library (PSMILe) API :

•Initialization: call prism_init_comp (comp_id, comp_name, ierr)

•Retrieval of component local communicator:call prism_get_localcomm (local_comm, ierr)

•Global grid definition (master process only)

call prism_write_grid (grd_name, nx, ny, lon, lat)call prism_write_corner (grd_name, nx, ny, nc, clon, clat)call prism_write_mask (grd_name, nx, ny, mask)call prism_write_areas (grd_name, nx, ny, area)



PRISM System Model Interface Library (PSMILe) API :

•Local partition definition (by each process in the global index space)

call prism_def_partition (part_id, ig_paral, ierr)

•Coupling or I/O field declaration (name, partition id, status, shape, type)

call prism_def_var (var_id, var_name, part_id, var_nbrdims, var_inout,

var_shape, var_type, ierr)

•End of definitioncall prism_enddef (ierr)



•Coupling or I/O field sending and receiving:in model time stepping loop depending on user’s specifications in namcouple:

•user’s defined source or target (end-point communication)•coupling or I/O sending or receiving at appropriate times•automatic averaging/accumulation•automatic writing of coupling restart file at end of run

call prism_put (var_id, time, var_array, ierr) call prism_get (var_id, time, var_array, ierr)

•Termination:call prism_terminate (ierr)

•Auxiliary routines:call prism_put_inquire (var_id, time, info)call prism_put_restart (var_id, time, info)

PRISM System Model Interface Library (PSMILe) API: :


Oasis3: coupled model configuration

•In text file namcouple, read by Oasis3 main process, and distributed to component model PSMILes at beginning of run:

•total run time

•component models

•number of coupling fields

•for each coupling field:

•source and target names (end-point communication) (var_name)

•grid acronym (grid_name)

•coupling and/or I/O status

•coupling or I/O period

•transformations/interpolations


Oasis3: communication

PSMILe based on MPI1 or MPI2 message passing

•End-point communication; one-to-one

A

A

A

B

B

B

•Direct communication between models with same grid and partitioning

A

A

A

file

•I/O functionality (automatic switch between coupled and forced mode): GFDL mpp_io library

A

A

A

O

O

O

O

Oasis3•Parallel communication between parallel models and Oasis3 interpolation process


Oasis3: interpolations/transformations

separate sequential processneighbourhood searchweight calculationinterpolation per se during the run

A

A

A

O

O

O

O

Oasis3

Oasis3

on 2D scalar fields only

• Interfacing with RPN Fast Scalar INTerpolator package

• nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids

• Interfacing with SCRIP1.4 library •nearest-neighbour, 1st and 2nd order conservative remapping for all grids • bilinear and bicubic interpolation for «logically-rectangular» grids

• Bilinear and bicubic interpolation for reduced atmospheric grids• Other spatial transformations: flux correction, merging, etc.• General algebraic operations


PRISM final coupler: Oasis4

Situation today:

•Higher resolution, parallel and scalable models

•Higher coupling frequencies desirable

•Higher number of models and (3D) coupling fields

•Massively parallel platformsNeed to optimise and parallelise the coupler

Oasis4 is composed of: •a Driver•a Transformer•a new PRISM System Model Interface Library


PRISM final coupler: Oasis4

Some keynotes: Developers:

Sophie Valcke (CERFACS) - Design, XML inputDamien Declat (CERFACS) - Driver, transformer, XML inputRené Redler (NEC CCRLE) - PSMIle, portability to Linux PC cluster Hubert Ritzdorf (NEC CCRLE) - Parallel search, transformer interaction Reiner Vogelsang (SGI) - I/O, portability to SGITh. Schoenemeyer (NEC HPCE) - Parallelisation, portability to NEC SX Jean Latour (Fujitsu ) - Portability to Fujitsu VPP and PC cluster

Public domain; open source license (LGPL)

All external libraries and modules used exist in public domain versions; vendor optimized versions may also exist:

• MPI1 and/or MPI2 • mpp_io; NetCDF/parallel NetCDF• libXML• SCRIP

Programming language: Fortran 90 and C



New PSMILe API:

•Application initialization (1 call/application) : call prism_init

(appl_name, ierr)

•Component application (1 call/component) :

call prism_init_comp (comp_id, comp_name, ierr)

•Retrieval of component local communicator (1 call/component):

call prism_get_localcomm

(comp_id, local_comm, ierr)

•Inquire SCC or SMIOCs persistent values:

call prism_get_persist (…)



•Definition of grid (3D)

call prism_def_grid (grd_id, grd_name, comp_id, grd_shape, type, ierr)

call prism_set_corners (grd_id, nbr_crnr, crnr_shape, crnr_array, ierr)

•Placement of scalar points, vector points, subgrids on the grid:

call prism_set_points(pt_id, pt_name, grd_id, pt_shape, pt_lon, pt_lat, pt_vert ,ierr)

call prism_set_vector(vct_id, vct_name, array_of_pt_ids,ierr)

call prism_set_subgrid(sgrd_id, sgrd_name, grd_id, n_sgrd, sgrd_shape, sgrd_array, ierr)

Function overloading to keep the interface concise and flexible



•Definition of grid cell dimension

call prism_set_scalefactors(grd_id, scale_shape, scale_array, ierr)

•Definition of grid cell mask

call prism_set_mask(msk_id, grd_id, msk_shape, msk_array, ierr)

•Definition of point local coordinate system angle

call prism_set_angle

(pt_id, angle_shape, angle_array, ierr)


prism_def_grid

Example

prism_set_mask

prism_set_points

prism_set_pointsprism_set_points

prism-set-corners


prism_set_vector



•Coupling or I/O field declaration

call prism_def_var(var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims,

var_shape, var_type, ierr)

•End of definitioncall prism_enddef (ierr)

•Coupling or I/O field sending and receiving:in model time stepping loop depending on user’s specifications in SMIOC:

•user’s defined source or target, component or file (end-point communication)•coupling or I/O sending or receiving at appropriate times •automatic averaging/accumulation•automatic writing of coupling restart file at end of run

call prism_put (var_id, date, date_bounds, var_array, info, ierr) call prism_get (var_id, date, date_bounds, var_array, info, ierr)


Application and component description (XML files):For each application (code):

one Application Description (AD):•possible number of processes•components included

For each component in the application:

one Potential Model Input and Output Description (PMIOD)

•component general characteristics: name, component simulated, …•code general characteristics: lab, documentation, contact, …•grid information: domain, resolution(s), grid type, …

•potential I/O or coupling variables:•local name, standard name •units, valid min and max •numerical type•associated grid and points •intent –input and/or output

•persistent variables: name, units, numerical type, possible values, …

Oasis4: coupled model configuration (1/2)


Coupled model configuration (XML files): The user produces

a Specific Coupling Configuration (SCC):• experiment and run start date and end date• start mode (MPI1, MPI2)• applications, components for each application• host(s), number of processes per host, ranks for each component

For each component, a Specific Model Input and Output Configuration (SMIOC)

• PMIOD component and code general characteristics• grid information: chosen resolution, … • I/O or coupling variables:

•local and standard name, units, valid min max, numerical type, grid •activated intent –input and/or output•source and/or target (component and/or file)•coupling or I/O dates•transformations/interpolations/combinations

•persistent variables: name, units, numerical type, chosen value, …

Oasis4: coupled model configuration (2/2)

Driver

TOCE

ATM

LAND

fileV4Definition Phase

OCE PMIODV1: out, metadata V1V2: in, metadata V2

ATM PMIODV1: in, metadata V1V2: out, metadata V2V3: out, metadata V3

LAND PMIODV3: in, metadata V3V4: in, metadata V4

ATM-LAND AD

OCE AD

OCE SMIOCV1 : to ATM, T1 to file V1V2 : from ATM, T2

userATM SMIOC V1 : from OCE, T1V2: to OCE, T2V3 : to LAND

user

LAND SMIOC V3 : from ATMV4 : from fileV4

user

Composition Phase

user

SCC ATM:...OCE:...LAND:...

user

Deployment Phase

V1V1

V2V2

V3

fileV1

V4


Oasis4: the Driver

•Manages a static coupling:

all applications are started directly in the run script (MPI1 mode) or launched by the Driver at the beginning of the run (MPI2 mode) for the whole run.

•Participates to communicator set-up

•Reads the SCC and SMIOcs information and distributes it to the component PSMILes

Driver


•More elaborate PSMILe based on MPI1 or MPI2

•Parallel communication including repartitioning:

•based on geographical description of the partitions

•parallel calculation of communication patterns in source PSMILe

OB

OB

OB

C

C

C

O1

O1

C

C

Same grid, different decomposition direct repartitioning:

• for each target point, parallel calculation of source matching point in source PSMILe

Oasis4: communication (1/3)

T

OB

OB

OB

C

C

C

O2

O2

C

C

Different grid and decomposition via Transformer:

• for each target point, parallel calculation of source interpolation neighbours in source PSMILe



x x x x x x x x x x x x x x x x x x x x

o o oo o oo o o

1/ Envelop exchange 2/ Detailed neighbourhood search

3/ EPIOS and EPIOT definition

Parallel calculation of communication patterns in source PSMILe

For each pair of source and target processes:

x x x x x x x x x

o o oo o o

x x x x x x x x x

o o oo o o

EPIOS1EPIOT1

x x x x x

o o oo o o

x xx xx x

x x x x x

o o oo o o

x xx xx xEPIOS2 EPIOT2


• parallel 3D neighbourhood search in each source process PSMILe

• new search algorithm: parallel multigrid technique, higher efficiency

• extraction of useful part of source field only• end-point communication • one-to-one or one-to-many; combination

• source data can be consumed by more than one target model or file and at a different frequencies


• parallel I/O (vector, bundles, vector bundles) : GFDL mpp_io, parNetCDF

A

A

A

file

single parallel

A

A

A

file1

file2

file3

distributed

A

A

A

file


PSMILe: •time transformations, local transformations•global conservation, combinations

Transformer:


T

OB

OB

OB

C

C

C

O2

O2

C

C•loops over PSMILe requests

•EPIOS and EPIOT definition (source PSMILe)

•For each EPIOT point, neighbours in EPIOS (source PSMILe)

=>Transformer calculates interpolation weights

•Prism_put of EPIOS (source PSMILe)=>Transformer performs interpolation per se

•Prism_get of EPIOT (target PSMILe) •well suited for MPI parallelisation


Different interpolation schemes:•nearest-neighbour, bilinear, bicubic, conservative remapping, …

On different types of fields

•1D-2D-3D scalar, vector, bundles, bundles of vectors

On different types of grids

•blockstructured (regular and irregular)

•unstructured

•static or dynamic



•Access of XML information •Coupling and I/O of n parallel applications with m components.•Coupling exchange with repartitionning, direct or through the (mono-process) Transformer •Interpolations:

•PSMILe (non-exact) parallel neighborhood search•3D nearest-neighbour

•I/O: •single and parallel mode

•Coupling and I/O exchange from one source to many targets •Local transformations (scatter, gather, add or mult scalar, statistics)

•Basic time transformation (average, accumul, min, max)

Oasis4: in the prototype (03/2004)


•Completion of PSMILe calendar routines (internal and external)

•PSMILe API for model access to SCC and SMIOC information

•Interpolation:•More schemes (conservative, 3D, etc.)•Exact parallel neighbour search

•Transformer parallelisation

•Field reduction, combination

•Full support of vector and bundle fields (I/O OK)

•I/O: distributed mode (parNetCDF)

•Adaptative grids

•Unstructured grids

Oasis4: still to be done


Conclusions

Calendar:•Oasis3_prism_2-1 available

•Oasis4 prototype available within days

•Oasis4 final PRISM version due 12/2004

Follow-on:•Oasis4 developments will go on after 12/2004 with PRISM sustained support

•FP6 funding (march 2005)?

Static vs dynamic couplers:•In the near and mid future, Oasis4 and Palm (Cerfacs dynamical coupler) will remain separate software

•Fusion may be considered when climate modelling and data assimilation communities will join


The end

oasis3 and oasis4 : the prism couplers g. berti, p. bourcier, a. caubel, d. declat, m.-a. foujols,...

Documents