Mesoscale & Microscale Meteorological Division / NCAR

ESMF and the Weather Research and Forecast Model

John Michalakes, Thomas HendersonMesoscale and Microscale Meteorology Division

National Center for Atmospheric Research

michalak@ucar.edu

Thomas BlackEnvironmental Modeling Center

NOAA / National Centers for Environmental Prediction

Weather Research and Forecast Model

Goals: Develop an advanced mesoscale forecast and assimilation system, and accelerate research advances into operations

• Large collaborative effort to develop next-generation community model with direct path to operations

• Advanced Software Architecture– Modular, flexible, extensible– Portable and efficient– Designed for HPC

• Applications– Atmospheric Research– Numerical Weather Prediction– Coupled modeling systems– Air quality research/prediction– High resolution regional climate

– prediction– Air quality research and prediction– Regional climate

Signatory Partners:NCAR Mesoscale and Microscale MeteorologyNOAA National Centers for Environmental PredictionNOAA Forecast Systems LaboratoryOU Center for the Analysis and Prediction of StormsU.S. Air Force Weather AgencyU.S. Naval Research LaboratoryFederal Aviation Administration

Additional Collaborators:U.S. Department of Defense HPCMPNOAA Geophysical Fluid Dynamics LaboratoryNASA GSFC Atmospheric Sciences DivisionNOAA National Severe Storms LaboratoryCAMS Chinese Meteorological AcademyEPA Atmospheric Modeling DivisionUniversity Community

Signatory Partners:NCAR Mesoscale and Microscale MeteorologyNOAA National Centers for Environmental PredictionNOAA Forecast Systems LaboratoryOU Center for the Analysis and Prediction of StormsU.S. Air Force Weather AgencyU.S. Naval Research LaboratoryFederal Aviation Administration

Additional Collaborators:U.S. Department of Defense HPCMPNOAA Geophysical Fluid Dynamics LaboratoryNASA GSFC Atmospheric Sciences DivisionNOAA National Severe Storms LaboratoryCAMS Chinese Meteorological AcademyEPA Atmospheric Modeling DivisionUniversity Community

+ DTC

Weather Research and Forecast Model

Goals: Develop an advanced mesoscale forecast and assimilation system, and accelerate research advances into operations

• Large collaborative effort to develop next-generation community model with direct path to operations

• Advanced Software Architecture– Modular, flexible, extensible– Portable and efficient– Designed for HPC

• Applications– Atmospheric Research– Numerical Weather Prediction– Coupled modeling systems– Air quality research/prediction– High resolution regional climate

– prediction– Air quality research and prediction– Regional climate

ARW solver

Physics Interfaces

Plug-compatible physicsPlug-compatible physics

Plug-compatible physicsPlug-compatible physics

Plug-compatible physics

NMM solver

Top-level Control,Memory Management, Nesting,

Parallelism, External APIs

med

iatio

ndr

iver

mod

el

Mesoscale & Microscale Meteorological Division / NCAR

ESMF Support

• WRFV2.2 fully interoperable as ESMF component– Runs stand-alone without ESMF, or– Fully integrated component in ESMF– Distributed with ESMF unit test coupled “data” ocean

• Infrastructure: – ESMF Time Manager using reference implementation or

shared implementation with CCSM

• Superstructure: “Wrapperized”– Top level init-run-finalize convention– ESMF conforms to “coupling as I/O” and is integrated with

WRF Registry

Mesoscale & Microscale Meteorological Division / NCAR

WRF Model Coupling• Extension of WRF I/O API

– MCEL (U. Southern Miss.)– MCT (Argonne NL)– ESMF

• Projects– WRF/ROMS,

WRF/ADCIRC/SWAN/LSOM– Very high resolution urban airflow

modeling (Fei Chen, NCAR; CFD Corp.)

– Coupled WRF/HYCOM Katrina simulation

MCEL MCT

ESMF Superstructure

ESMF

WRF/HYCOM Coupling (ESMF)

• WRF and HYCOM two-way coupled through ESMF to improve modeling of hurricane intensification

• Synchronization is through successive calls to components by ESMF top-level driver

• Status:– WRF an ESMF component

(NCAR), • Init-run-finalize at top level• Import/export through WRF I/O

API• ESMF Time Manager• WRF I/O in ESMF

– HYCOM-ESMF (NRL)

WR

F I

/O A

PI

WRF component

ESMF WRF Component Interface

HYCOM component

ESMF WRF Component Interface

ESMF Superstructure

ESMF Infrastructure

WRF export state: Surface winds & air temps Precipitation Radiation fluxesWRF import state: SST, Roughness length

HYCOM export state: SSTsHYCOM import state: Winds Precipitation Radiation fluxes

execution controlregridding

…

Mesoscale & Microscale Meteorological Division / NCAR

WRF/LIS Coupling through ESMF

• LIS = Land Information System– NASA GSFC (lis.gsfc.nasa.gov)– Incorporates CLM, Noah, and VIC land surface models

• Initial WRF-LIS prototype by Sujay Kumar (GSFC)– Coupled via WRF surface driver using ESMF– All ESMF objects manufactured by LIS

• Planned– WRF provides ESMF_State objects to LIS– Extend to allow different grids

Mesoscale & Microscale Meteorological Division / NCAR

Interoperability

• Supported metadata formats: NetCDF+parallel, HDF+parallel, GRIB[1&2]

• Participating in continuing refinement of Climate and Forecast metadata convention– Participate in GOESSP meetings to discuss

conventions (go-essp.gfdl.noaa.gov)– Provide feedback for Balaji’s “Gridspec” proposal (

www.gfdl.noaa.gov/~vb/gridstd/gridstd.html)

Mesoscale & Microscale Meteorological Division / NCAR

Interoperability

• Share goals with Earth System Curator– Similar approaches may apply to forecast V&V,

managing ensembles, etc.• Earth System Curator-Numerical Model Metadata (NMM)

standard– http://ncas-cms.nerc.ac.uk/NMM/index.php – Full specification of an “experiment”

• Automatic testing of component compatibility

Mesoscale & Microscale Meteorological Division / NCAR

Dynamic Load Balancing

• Key obstacle for Petascale; opportunity for new approaches, e.g. PGAS languages

• Instrument WRF using PAPI• Generated high-resolution traces

over 8 day period in July ’05• Characterize dynamic load from

microphysics as functions of– Decomposition

– Sizes, trajectories, other characteristics of load features

Workload Characterization of Physics-Induced Load Imbalance in WRF (draft NCAR TM, 2007)

Mesoscale & Microscale Meteorological Division / NCAR

Dynamic Load Balancing

 Load Imbalance vs Tile Size

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

5 10 15 20 25 30 35 40

Tile Size

1 -

mea

n /

max

.

max

average

min

sd

Figure 6: Maximum, average, minimum, and standard-deviation of load imbalance over eight-day measurement period as function of subdomain size.

Workload Characterization of Physics-Induced Load Imbalance in WRF (draft NCAR TM, 2007)

• Key obstacle for Petascale; opportunity for new approaches, e.g. PGAS languages

• Instrument WRF using PAPI• Generated high-resolution traces

over 8 day period in July ’05• Characterize dynamic load from

microphysics as functions of– Decomposition

– Sizes, trajectories, other characteristics of load features

Mesoscale & Microscale Meteorological Division / NCAR

Plans

• Convergence of ESMF and WRF software– WRF as ESMF component (current)– Support for additional infrastructure elements– Support for ESMF components coupled in WRF

call tree– Continue to follow, participate in data-

interoperability standardization (C.F., Curator)