www.esmf.ucar.edu

Earth System Modeling Infrastructure

Cecelia DeLuca/ESMF-NCAR

March 31-April 1, 2009CHyMP Meeting

www.esmf.ucar.edu

Outline

• Elements of interoperability platforms• Integrating across elements• Summary

www.esmf.ucar.edu

Elements of interoperability platforms 1. Tight coupling tools and interfaces

- hierarchical and peer component relationships- frequent, high volume transfers on high performance computers

2. Loose coupling tools and interfaces- generally peer-peer component relationships- lower volume and infrequent transfers on desktop and distributed systems

3. Science gateways- browse, search, and distribution of model components, models, and datasets- visualization and analysis services- workspaces and management tools for collaboration

4. Metadata conventions and ontologies- ideally, with automated production of metadata from models

5. Governance- coordinated and controlled evolution of systems

www.esmf.ucar.edu

Tight coupling tools and interfacesExamples:• Earth System Modeling Framework (ESMF) - NASA, NOAA,

Department of Defense, community weather and climate models, U.S. operational numerical weather prediction centers (HPC focus)http://www.esmf.ucar.edu

• Flexible Modeling System (FMS) – NOAA precursor to ESMF, still used at the Geophysical Fluid Dynamics Laboratory for climate modelinghttp://www.gfdl.noaa.gov/fms/

• Space Weather Modeling Framework (SWMF) – NASA-funded, used at the University of Michigan for space weather prediction

www.esmf.ucar.edu

How coupling tools work:

• Users wrap their native data in framework data structures• Users adopt standard calling interfaces for a set of methods that

enable data exchange between components• Development toolkits help users with routine functions (regridding,

time management, etc.)

www.esmf.ucar.edu

ESMF: Standard interfaces

• Three ESMF component methods: Initialize, Run, and Finalize (I/R/F)

• Each can have multiple phases• Users register their native I/R/F

methods with an ESMF Component

• Small set of arguments:

Child GridComp “Atmosphere”

Parent GridComp “Hurricane Model”

Finalize

Child GridComp “Ocean”

Finalize

Child CplComp “Atm-Ocean Coupler”

Finalize

Call Initialize Call FinalizeCall Run

Initialize Run Finalize

Initialize

Initialize

Initialize

Run

Run

Run

AppDriver (“Main”)

Call Initialize Call FinalizeCall Run

call ESMF_GridCompRun (myComp, importState, exportState,clock, phase, blockingFlag, rc)

www.esmf.ucar.edu

ESMF: Distributed data representation 1. Representation in index space (Arrays)• Simple, flexible multi-dimensional

array structure• Regridding via sparse matrix multiply with

user-supplied interpolation weights• Scalable to 10K+ processors - no global

information held locally

2. Representation in physical space (Fields)• Built on Arrays + some form of Grid• Grids are: logically rectangular, unstructured mesh, or observational data streams• Regridding via parallel on-line interpolation weight generation, bilinear or higher order

options• Intrinsically holds significant amounts of metadata -

dynamic, usable for multiple purposes, limited annotation required

Supported Array distributions

www.esmf.ucar.edu

ESMF: Coupling options

• Generally single executable for simpler deployment• Push mode of data communication is very efficient• Coupling communications can be set up and called in a coupler, or called directly

from within components (for I/O, data assimilation)

GEOS-5

surface fvcore gravity_wave_drag

history agcm

dynamics physics

chemistry moist_processes radiation turbulence

infrared solar lake land_ice data_ocean land

vegetation catchment

coupler

coupler coupler

coupler

coupler

coupler

coupler

• Hierarchical components for organization into sub-processes

• Recursive components for nesting higher resolution regions

• Coupling across C/C++and Fortran

• Ensemble management

ESMF-based hierarchical structure of GEOS-5 atmospheric GCM

www.esmf.ucar.edu

ASMM Run Time Comparison on XT4 (w/o -N1)

0

1

10

100

1000

4 8 16 32 64 128

256

512

1024

2048

4096

8192

1638

4

Number of Processors

Tim

e (

mse

cs)

3D Array

Bundle

3D Array (-N1)

Bundle (-N1)

ESMF: Performance portability• ESMF is highly performance portable, low (<5%) overhead• 3000+ regression tests run on 30+ platform/compiler combinations nightly

See http://www.esmf.ucar.edu/download/platforms• Newer ports include native Windows, Solaris• Using TeraGrid Build and Test Service to simplify regression testing

Performance at the petascale…

Scaling of the ESMF sparse matrix multiply, used in regridding transformations, out to 16K processors.(ESMF v3.1.0rp2)

Plot from Peggy Li, NASA/JPLTested on ORNL XT4,-N1 means 1 core per node.

msec

ASMM Run-Time Comparison

www.esmf.ucar.edu

ESMF: Higher order interpolation techniques in CCSM

Interp.noise

Interpolation noise in the derivative of the zonal wind stress

grid index in latitudinal direction

• ESMF higher order interpolation weights were used to map from a 2-degree Community Atmospheric Model (CAM) grid to a POP ocean grid (384x320, irregularly spaced)

• 33% reduction in noise globally in quantity critical for ocean circulation compared to previous bilinear interpolation approach

• ESMF weights are now the CCSM default

Black = bilinearRed = higher-orderESMF v3.1.1Green = higher order ESMF v4.0.0

www.esmf.ucar.edu

POP

UCLA AGCM

WRF

NCOM HYCOM

CICE

pWASH123 ADCIRC

ROMS

CICE ice

POP Ocean

CCSM4

COAMPS SWAN

NMM-B Atm PhysNMM-B Atm DynamicsNEMS

NMM History

SWMF

MITgcm Atm MITgcm Ocean

MITgcm

GFS Atm Phys GFS Atm DynamicsGFS

GFS I/O

Land Info SystemFV Cub Sph DycoreGEOS-5 GWD GEOS-5 FV Dycore

GEOS-5 Atm Dynamics

GEOS-5

GSI

MOM4

GEOS-5 Moist Proc

GEOS-5 Turbulence

GEOS-5 LW Rad GEOS-5 Solar Rad

GEOS-5 Radiation

GEOS-5 Aeros Chem

GOCART

Strat Chem

Param Chem

GEOS-5 Atm Chem

GEOS-5 Ocean BiogeoGEOS-5 Salt WaterPoseidon GEOS-5 Data Ocean

GEOS-5 OGCM

GEOS-5 Topology

GEOS-5 Land Ice

GEOS-5 Lake GEOS-5 Veg Dyn GEOS-5 Catchment

GEOS-5 Land

GEOS-5 Surface

GEOS-5 Atm PhysicsGEOS-5 Hiistory

ESMF: Model map

NOAADepartment of DefenseUniversityNASADepartment of EnergyNational Science Foundation

ESMF coupling completeESMF coupling in progress

Component (thin lines)Model (thick lines)

LegendOvals show ESMF components and modelsthat are at the working prototype level orbeyond.

Tracer Advection

HAF

GAIM

CLM

Ice sheet

Dead ocean

Dead ice

Data ocean

Data ice

Dead land Data land

Stub ocean

Stub ice

Stub landFIM

Dead atm Data atm

www.esmf.ucar.edu

Loose coupling tools and interfacesExamples:• OpenMI

http://www.openmi.org• Web service approaches

Coupling options:• Generally multiple executable• Pull mode of data communication simple but not efficient

(ask for a data point based on coordinates)• Generally peer-peer component relationships• Coupling across multiple computer languages (Python, Java, C++,

etc.)

www.esmf.ucar.edu

Science gateways – access centersExamples:• Earth System Grid (ESG) – DOE, NCAR, NOAA support, used to

distribute Intergovernmental Panel on Climate Change data and for climate researchhttp://www.earthsystemgrid.org

• Hydrologic Information System (HIS) - NSF funded, used to enhance access to data for hydrologic analysishttp://his.cuahsi.org

• Object Modeling System (OMS) - USDA effort, used for agricultural modeling and analysishttp://javaforge.com/project/1781

www.esmf.ucar.edu

Metadata conventions and ontologiesExamples:• Climate and Forecast (CF) conventions - spatial and temporal

properties of fields used in weather and climatehttp://cf-pcmdi.llnl.gov

• METAFOR Common Information Model (CIM) – large EU-funded project, climate model component structure and properties (including technical and scientific properties) http://metaforclimate.eu

• WaterML – Schema for hydrologic data developed by the Consortium of Universities for the Advancement of Hydrologic Science (CUAHSI)http://his.cuahsi.org/wofws.html

www.esmf.ucar.edu

Governance

Pervasive issue in community modelingDivergent effects of• Multiple institutions• Geographic dispersion• Multiple domains of interest (working groups)

Must be balanced by strong integration body - strategies:• Meets frequently enough to affect routine development (quarterly)• Meets virtually to get sufficient representation• Includes user and other stakeholder representatives• Authorized to prioritize and set development schedule• Supported by web-based management tools

www.esmf.ucar.edu

Integrating across interoperability elementsExamples from the Curator project (NSF and NASA)• Automated output of CF and CIM XML schema from ESMF

(tight coupling + ontology) • Ingest of ESMF-generated schema into ESG, propagation

into tools for search, browse, inter-comparison and distribution of model components and models(tight coupling + ontology + science gateway)

• Implementation of dataset “trackback” in ESG that connects datasets with detailed information about the models used to create the data(tight coupling + ontology + science gateway)

• Implementation of personal and group workspaces in ESG(science gateway + governance)

www.esmf.ucar.edu

Integrating across interoperability elements(cont.)• Translation of ESMF interfaces into web services to enable invocation

of ESMF applications from a science gateway, and enable data and metadata from the run to be stored back to the gateway(tight coupling + loose coupling + science gateway + ontology, new TeraGrid funding)

ESMFinterface

Web serviceinterface

Tightlycoupled HPCcomponents

Looselycoupledcomponents

Issue of switchfrom push to pulldata interactions…

www.esmf.ucar.edu

Screenshot: Component trackback

www.esmf.ucar.edu

Screenshot: Faceted search

www.esmf.ucar.edu

Summary

• Cross-domain interoperability platforms have multiple elements• Many of these elements already exist• Integration activities (such as Earth System Curator) are the next

focus

Image courtesy of Rocky Dunlap, Georgia Institute of technology