monday january 26, 2004
DESCRIPTION
Welcome To the Inaugural Meeting of the WRF Software Training and Documentation Team Jan. 26-28, 2004 NCAR, MMM Division. Monday January 26, 2004. Introduction Software Overview WRF Software Tutorial, June 2003 Data and data structures Parallel infrastructure. Introduction. History - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/1.jpg)
Welcome To the Inaugural Meeting of the
WRF Software Training and Documentation Team
Jan. 26-28, 2004
NCAR, MMM Division
![Page 2: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/2.jpg)
Monday January 26, 2004
• Introduction
• Software Overview
• WRF Software Tutorial, June 2003
• Data and data structures
• Parallel infrastructure
![Page 3: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/3.jpg)
Introduction
• History– Requirements emphasize flexibility over a
range of platforms, applications, users– WRF develops rapidly. First released Dec
2000; Last beta release, 1.3, in May 2003. Official 2.0 release coming in May 2004
– Circa 2003: "Arcane" used to describe WRF: • Adj. Known or understood by only a few.
Mysterious.
![Page 4: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/4.jpg)
Introduction
• Purpose of WRF Tiger Team effort– Extend knowledge of WRF software to wider
base of software developers– Create comprehensive developer document– Team approach to both objectives– Streamlining and code improvement as
byproducts in the coming months but not the subject for this meeting
![Page 5: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/5.jpg)
Introduction
• This meeting– Review of WRF software structure and function:
• Phenomenological – this is the code as it exists• Incomplete –time to prepare and present is limiter, but• What we're looking for now is a roadmap through the code for
producing the comprehensive documentation
– Develop an outline for the developer documentation– Writing assignments and work plan over next 9 months
![Page 6: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/6.jpg)
Some terms• WRF Architecture – scheme of software layers
and interface definitions• WRF Framework – the software infrastructure,
also "driver layer" in the WRF architecture• WRF Model Layer – the computational routines
that are specifically WRF• WRF Model – a realization of the WRF
architecture comprising the WRF model layer with some framework
• WRF – a set of WRF architecture-compliant applications, of which the WRF Model is one
![Page 7: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/7.jpg)
WRF Software Overview
![Page 8: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/8.jpg)
Weather Research and Forecast ModelGoals: Develop an advanced mesoscale forecast
and assimilation system, and accelerate research advances into operations
12km WRF simulation of large-scale baroclinic cyclone,
Oct. 24, 2001
![Page 9: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/9.jpg)
WRF Software Requirements…
• Fully support user community's needs: nesting, coupling, contributed physics code and multiple dynamical cores – but keep it simple
• Support every computer but make sure scaling and performance is optimal on the computer we use
• Leverage community infrastructure, computational frameworks, contributed software, but please no opaque code
• Implement by committee of geographically remote developers• Adhere to union of all software process models• Fully test, document, support• Free
;-)
![Page 10: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/10.jpg)
WRF Software Requirements (for real)
Goals• Community Model
• Good performance
• Portable across a range of architectures
• Flexible, maintainable, understandable
• Facilitate code reuse
• Multiple dynamics/ physics options
• Run-time configurable
• Nested
• Package independent
Aspects of Design• Single-source code
• Fortran90 modules, dynamic memory, structures, recursion
• Hierarchical software architecture
• Multi-level parallelism
• CASE: Registry
• Package-neutral APIs– I/O, data formats– Communication
• Scalable nesting/coupling infrastructure
![Page 11: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/11.jpg)
Aspects of WRF Software Design
![Page 12: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/12.jpg)
Aspects of WRF Software Design
![Page 13: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/13.jpg)
Model Coupling
![Page 14: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/14.jpg)
ESMF
![Page 15: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/15.jpg)
Performance
![Page 16: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/16.jpg)
Structural Aspects
• Directory Structure and relationship to Software Hierarchy
• File nomenclature and conventions
• Use Association
![Page 17: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/17.jpg)
Directory Structure
![Page 18: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/18.jpg)
WRF Model Directory Structure
page 5, WRF D&I Document
drivermediation
model
![Page 19: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/19.jpg)
WRF File Taxonomy and NomenclatureModule files
14 Driver frame/module_* .F WRF framework (driver layer)1 Driver frame/module_state_description.F registry generated framework file
16 Model dyn_em/module_*.F em core-specific model layer18 Model dyn_nmm/module_* .F nmm core-specific model layer11 Model share/module_*. F non-core specific model layer22 Model phys/module_pp_*. F physics modules, where pp is kind of physics2 Model phys/module_*.F misc physics routines5 Mediation phys/module_*_ driver.F physics drivers
Non-module Fortran Source7 Driver main/*.F main programs (1 wrf and 6 preprocs)3 Driver frame/*.c C-language routines in the WRF framework4 Mediation dyn_em/*. F em core-specific routines (includes solver)
12 Mediation dyn_nmm/*. F nmm core-specific routines (includes solver)5 Mediation share/mediation_*. F mediation layer5 Mediation share/something. F mediation layer and miscellaneous
Include filesinc/*. inc registry generated includesinc/*. h io api definitions, autogenerated from build
Others13 Makefile */Makefile build mechanism2 configure and compile scripts build mechanism9 tools/*.c source for registry program1 tools/regtest.csh a regression tester for WRF model
Externals7 External external/* external package directories
![Page 20: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/20.jpg)
Module Conventions and USE Association
• Modules are named module_something
• Name of file containing module is module_something.F
• If a module includes an initialization routine, that routine should be named init_module_something()
• Typically: – Driver and model layers are made
up of modules,– Mediation layer is not (rather, bare
subroutines), except for physics drivers in phys directory
– Gives benefit of modules while avoiding cycles in the use association graph
MODULE module_thisMODULE module_that
…
USE module_thisUSE module_that
USE module_whatchaUSE module_macallit
…
MODULE module_whatchaMODULE module_macallit
USE module_thisUSE module_that
…
driver
mediation
model
![Page 21: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/21.jpg)
WRF S/W Tutorial, June 2003
![Page 22: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/22.jpg)
Tutorial Presentation (click here)
• Parallel Infrastructure
• Registry details
• I/O architecture and mechanism
• Example of coding new package into framework
![Page 23: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/23.jpg)
Data and Data Structures
![Page 24: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/24.jpg)
Session 3: Data Structures
• Overview
• Representation of domain
• Special representations– Lateral Boundary Conditions– 4D Tracer Arrays
![Page 25: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/25.jpg)
Data Overview
• WRF Data Taxonomy– State data– Intermediate data type 1 (I1)– Intermediate data type 2 (I2)– Heap storage (COMMON or Module data)
![Page 26: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/26.jpg)
State Data
• Persist for the duration of a domain• Represented as fields in domain data structure• Arrays are represented as dynamically allocated
pointer arrays in the domain data structure• Declared in Registry using state keyword• Always memory dimensioned; always thread
shared• Only state arrays can be subject to I/O and
Interprocessor communication
![Page 27: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/27.jpg)
I1 Data
• Data that persists for the duration of 1 time step on a domain and then released
• Declared in Registry using i1 keyword• Typically automatic storage (program stack) in
solve routine• Typical usage is for tendency arrays in solver• Always memory dimensioned and thread
shared• Typically not communicated or I/O
![Page 28: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/28.jpg)
I2 Data
• I2 data are local arrays that exist only in model-layer subroutines and exist only for the duration of the call to the subroutine
• I2 data is not declared in Registry, never communicated and never input or output
• I2 data is tile dimensioned and thread local; over-dimensioning within the routine for redundant computation is allowed – the responsibility of the model layer programmer– should always be limited to thread-local data
![Page 29: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/29.jpg)
Heap Storage
• Data stored on the process heap is not thread- safe and is generally forbidden anywhere in WRF– COMMON declarations– Module data
• Exception: If the data object is:– Completely contained and private within a Model
Layer module, and– Set once and then read-only ever after, and– No decomposed dimensions.
![Page 30: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/30.jpg)
Grid Representation in Arrays
• Increasing indices in WRF arrays run– West to East (X, or I-dimension)– South to North (Y, or J-dimension)– Bottom to Top (Z, or K-dimension)
• Storage order in WRF is IKJ but this is a WRF Model convention, not a restriction of the WRF Software Framework
![Page 31: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/31.jpg)
Grid Representation in Arrays
• The extent of the logical or domain dimensions is always the "staggered" grid dimension. That is, from the point of view of a non-staggered dimension, there is always an extra cell on the end of the domain dimension.
![Page 32: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/32.jpg)
Grid Indices Mapped onto Array Indices (C-grid example)
m1,4 m2,4 m3,4 m4,4
m1,3 m2,3 m3,3 m4,3
m1,2 m2,2 m3,2 m4,2
m1,1 m2,1 m3,1 m4,1
u1,4 u2,4 u3,4 u4,4
u1,3 u2,3 u3,3 u4,3
u1,2 u2,2 u3,2 u4,2
u1,1 u2,1 u3,1 u4,1
u4,5
u5,3
u5,2
u5,1
v1,4 v2,4 v3,4 v4,4
v1,3 v2,3 v3,3 v4,3
v1,2 v2,2 v3,2 v4,2
v1,1 v2,1 v3,1 v4,1
v1,5 v2,5 v3,5 v4,5
jds = 1
jde = 5
ids = 1 ide = 5
Computation overmass points runsonly ids..ide-1and jds..jde-1
Likewise, verticalcomputation over unstaggered fieldsrun kds..kde-1
![Page 33: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/33.jpg)
LBC Arrays
• State arrays, declared in Registry using the b modifier in the dimension field of the entry
• Store specified forcing data on domain 1, or forcing data from parent on a nest
• All four boundaries are stored in the array; last index is over:
P_XSB (western)P_XEB (eastern)P_YSB (southern)P_YEB (northern)
These are defined in module_state_description.F
![Page 34: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/34.jpg)
LBC Arrays• LBC arrays are declared as follows:
em_u_b(max(ide,jde),kde,spec_bdy_width,4)
• Globally dimensioned in first index as the maximum of x and y dimensions
• Second index is over vertical dimension• Third index is the width of the boundary (namelist)• Fourth index is which boundary
• Note: LBC arrays are globally dimensioned• not fully dimensioned so still scalable in memory• preserves global address space for dealing with LBCs• makes input trivial (just read and broadcast)
![Page 35: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/35.jpg)
unused
LBC Arrays
unused
P_YEB
P_YSB
A Given Domain
P_Y
EB
P_X
EB
jds
jde
ids ide
spec_bdy_width
![Page 36: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/36.jpg)
unused
LBC Arrays
unused
P_YEB
P_YSB
P_Y
EB
P_X
EB
jds
jde
ids ide
A given subdomain that includes adomain boundary
spec_bdy_width
![Page 37: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/37.jpg)
Four Dimensional Tracer Arrays
• State arrays, used to store arrays of 3D fields such as moisture tracers, chemical species, ensemble members, etc.
• First 3 indices are over grid dimensions; last dimension is the tracer index
• Each tracer is declared in the Registry as a separate state array but with f and optionally also t modifiers to the dimension field of the entry
• The field is then added to the 4D array whose name is given by the use field of the Registry entry
![Page 38: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/38.jpg)
Four Dimensional Tracer Arrays
• Fields of a 4D array are input and output separately and appear as any other 3D field in a WRF dataset
• The extent of the last dimension of a tracer array is from PARAM_FIRST_SCALAR to num_tracername – Both defined in Registry-generated
frame/module_state_description.F– PARAM_FIRST_SCALAR is a defined constant (2)– Num_tracername is computed at run-time in
set_scalar_indices_from_config (module_configure)– Calculation is based on which of the tracer arrays are associated
with which specific packages in the Registry and on which of those packages is active at run time (namelist.input)
![Page 39: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/39.jpg)
Four Dimensional Tracer Arrays
• Each tracer index (e.g. P_QV) into the 4D array is also defined in module_state_description and set in set_scalar_indices_from_config
• Code should always test that a tracer index greater than or equal to PARAM_FIRST_SCALAR before referencing the tracer (inactive tracers have an index of 1)
• Loops over tracer indices should always run from PARAM_FIRST_SCALAR to num_tracername -- EXAMPLE
![Page 40: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/40.jpg)
Parallel Infrastructure
![Page 41: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/41.jpg)
Parallel Infrastructure
• Distributed memory parallelism– Some basics– API
• module_dm.F routines
• Registry interface (gen_comms.c)
– Data decomposition– Communications
• Shared memory parallelism– Tiling– Threading directives– Thread safety
![Page 42: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/42.jpg)
Some Basics on DM Parallelism
• Principal types of explicit communication– Halo exchanges– Periodic boundary updates– Parallel transposes– Special purpose scatter gather for nesting
• Also– Broadcasts– Reductions (missing, using MPI directly)– Patch-to-global and global-to-patch– Built in I/O server mechanism
![Page 43: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/43.jpg)
Some Basics on DM Parallelism
• All DM comm operations are collective• Semantics for specifying halos exchanges,
periodic bdy updates, and transposes allow message agglomeration (bundling)
• Halos and periods allow fields to have varying width stencils within the same operation
• Efficient implementation is up to the external package implementing communications
![Page 44: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/44.jpg)
DM Comms API
• External package provides a number of subroutines in module_dm.F
Click here for partial API specification
• Actual invocation of halos, periods, transposes provided by a specific external package is through #include files in the inc directory. This provides greater flexibility and latitude to the implementer than a subroutine interface
• Package implementer may define comm-invocation include files manually or they can be generated automatically by the Registry by providing a routine external/package/gen_comms.c for inclusion in the Registry program
![Page 45: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/45.jpg)
A few notes on RSL implementation
• RSL maintains descriptors for domains and the operations on the domains
• An operation such as a halo exchange is a collection of logical "messages", one per point on the halo's stencil
• Each message is a collection of fields that should be exchanged for that point
• RSL stores up this information in tables then compiles an efficient communication schedule the first time the operation is invoked for a domain
24 pt stencil
msg1{ u, v }
msg2{ t, w, ps }
. . .
![Page 46: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/46.jpg)
Example HALO_EM_D2_5
Defined in Registry
halo HALO_EM_D2_5 dyn_em 48:u_2,v_2,w_2,t_2,ph_2;\
24:moist_2,chem_2;\
4:mu_2,al
USED in dyn_em/solve_em.F
#ifdef DM_PARALLEL IF ( h_mom_adv_order <= 4 ) THEN # include "HALO_EM_D2_3.inc" ELSE IF ( h_mom_adv_order <= 6 ) THEN # include "HALO_EM_D2_5.inc" ELSE WRITE(wrf_err_message,*)'solve_em: invalid h_mom_adv_order ' CALL wrf_error_fatal (TRIM(wrf_err_message)) ENDIF # include "PERIOD_BDY_EM_D.inc" # include "PERIOD_BDY_EM_MOIST2.inc" # include "PERIOD_BDY_EM_CHEM2.inc" #endif
![Page 47: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/47.jpg)
Example HALO_EM_D2_5
Defined in Registry
halo HALO_EM_D2_5 dyn_em 48:u_2,v_2,w_2,t_2,ph_2;\
24:moist_2,chem_2;\
4:mu_2,al
!STARTOFREGISTRYGENERATEDINCLUDE 'inc/HALO_EM_D2_5.inc'!! WARNING This file is generated automatically by use_registry! using the data base in the file named Registry.! Do not edit. Your changes to this file will be lost.!IF ( grid%comms( HALO_EM_D2_5 ) == invalid_message_value ) THEN CALL wrf_debug ( 50 , 'set up halo HALO_EM_D2_5' ) CALL setup_halo_rsl( grid ) CALL reset_msgs_48pt CALL add_msg_48pt_real ( u_2 , (glen(2)) ) CALL add_msg_48pt_real ( v_2 , (glen(2)) ) CALL add_msg_48pt_real ( w_2 , (glen(2)) ) CALL add_msg_48pt_real ( t_2 , (glen(2)) ) CALL add_msg_48pt_real ( ph_2 , (glen(2)) ) if ( P_qv .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qv), glen(2) ) if ( P_qc .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qc), glen(2) ) if ( P_qr .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qr), glen(2) ) if ( P_qi .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qi), glen(2) ) if ( P_qs .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qs), glen(2) ) if ( P_qg .GT. 1 ) CALL add_msg_24pt_real ( moist_2 ( grid%sm31,grid%sm32,grid%sm33,P_qg), glen(2) ) CALL add_msg_4pt_real ( mu_2 , 1 ) CALL add_msg_4pt_real ( al , (glen(2)) ) CALL stencil_48pt ( grid%domdesc , grid%comms ( HALO_EM_D2_5 ) )ENDIF
CALL rsl_exch_stencil ( grid%domdesc , grid%comms( HALO_EM_D2_5 ) )
![Page 48: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/48.jpg)
Notes on Period Communication
m1,4 m2,4 m3,4 m4,4
m1,3 m2,3 m3,3 m4,3
m1,2 m2,2 m3,2 m4,2
m1,1 m2,1 m3,1 m4,1
u1,4 u2,4 u3,4 u4,4
u1,3 u2,3 u3,3 u4,3
u1,2 u2,2 u3,2 u4,2
u1,1 u2,1 u3,1 u4,1
u4,5
u5,3
u5,2
u5,1
v1,4 v2,4 v3,4 v4,4
v1,3 v2,3 v3,3 v4,3
v1,2 v2,2 v3,2 v4,2
v1,1 v2,1 v3,1 v4,1
v1,5 v2,5 v3,5 v4,5
updatingMass Pointperiodicboundary
![Page 49: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/49.jpg)
Notes on Period Communication
m1,4 m2,4 m3,4 m4,4
m1,3 m2,3 m3,3 m4,3
m1,2 m2,2 m3,2 m4,2
m1,1 m2,1 m3,1 m4,1
u1,4 u2,4 u3,4 u4,4
u1,3 u2,3 u3,3 u4,3
u1,2 u2,2 u3,2 u4,2
u1,1 u2,1 u3,1 u4,1
u4,5
u5,3
u5,2
u5,1
v1,4 v2,4 v3,4 v4,4
v1,3 v2,3 v3,3 v4,3
v1,2 v2,2 v3,2 v4,2
v1,1 v2,1 v3,1 v4,1
v1,5 v2,5 v3,5 v4,5
updatingMass Pointperiodicboundary
m4,4
m4,3
m4,2
m4,1
m1,4
m1,3
m1,2
m1,1
![Page 50: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/50.jpg)
Notes on Period Communication
m1,4 m2,4 m3,4 m4,4
m1,3 m2,3 m3,3 m4,3
m1,2 m2,2 m3,2 m4,2
m1,1 m2,1 m3,1 m4,1
u1,4 u2,4 u3,4 u4,4
u1,3 u2,3 u3,3 u4,3
u1,2 u2,2 u3,2 u4,2
u1,1 u2,1 u3,1 u4,1
u1,5
u1,3
u1,2
u1,1
v1,4 v2,4 v3,4 v4,4
v1,3 v2,3 v3,3 v4,3
v1,2 v2,2 v3,2 v4,2
v1,1 v2,1 v3,1 v4,1
v1,5 v2,5 v3,5 v4,5
updatingU Staggered periodicboundary
note: replicated
![Page 51: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/51.jpg)
Notes on Period Communication
m1,4 m2,4 m3,4 m4,4
m1,3 m2,3 m3,3 m4,3
m1,2 m2,2 m3,2 m4,2
m1,1 m2,1 m3,1 m4,1
u1,4 u2,4 u3,4 u4,4
u1,3 u2,3 u3,3 u4,3
u1,2 u2,2 u3,2 u4,2
u1,1 u2,1 u3,1 u4,1
u1,5
u1,3
u1,2
u1,1
v1,4 v2,4 v3,4 v4,4
v1,3 v2,3 v3,3 v4,3
v1,2 v2,2 v3,2 v4,2
v1,1 v2,1 v3,1 v4,1
v1,5 v2,5 v3,5 v4,5
updatingU Staggered periodicboundary
u4,4
u4,3
u4,2
u4,1
u2,4
u2,3
u2,2
u2,1
![Page 52: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/52.jpg)
Welcome To the Inaugural Meeting of the
WRF Software Training and Documentation Team
Jan. 26-28, 2004
NCAR, MMM Division
![Page 53: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/53.jpg)
Tuesday, January 27, 2004
• Detailed code walk-through
• I/O
• Misc. Topics– Registry– Error handling– Time management– Build mechanism
![Page 54: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/54.jpg)
Detailed WRF Code Walkthrough
![Page 55: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/55.jpg)
Detailed Code Walkthrough
• The walkthrough was conducted using the following set of Notes
• The WRF Code Browser was used to peruse the code and dive down at various points
• The walkthrough began with the main/wrf.F routine (when you bring up the browser, this should be in the upper right hand frame; if not, click the link WRF in the lower left hand frame, under Programs)
![Page 56: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/56.jpg)
I/O
![Page 57: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/57.jpg)
I/O
• Concepts
• I/O Software Stack
• I/O and Model Coupling API
![Page 58: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/58.jpg)
WRF I/O Concepts
• WRF model has multiple input and output streams that are bound a particular format at run time
• Different formats (NetCDF, HDF, binary I/O) are implemented behind a standardized WRF I/O API
• Lower levels of the WRF I/O software stack allow expression of a dataset open as a two-stage operation: OPEN BEGIN and then OPEN COMMIT– Between the OPEN BEGIN and OPEN COMMIT the program performs
the sequence of writes that will constitute one frame of output to "train" the interface
– An implementation of the API is free to use this information for optimization/bundling/etc. or ignore it
• Higher levels of the WRF I/O software stack provide a BEGIN/TRAIN/COMMIT form of an OPEN as a single call
![Page 59: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/59.jpg)
I/O Software Stack
• Domain I/O
• Field I/O
• Package-independent I/O API
• Package-specific I/O API
![Page 60: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/60.jpg)
Domain I/O
• Routines in share/module_io_domain.F– High level routines that apply to operations on a
domain and a stream• open and define a stream for writing in a single call that
contains the OPEN FOR WRITE BEGIN, the series of "training writes" to a dataset, and the final OPEN FOR WRITE COMMIT
• read or write all the fields of a domain that make up a complete frame on a stream (as specified in the Registry) with a single call
• some wrf-model specific file name manipulation routines
![Page 61: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/61.jpg)
Field I/O
• Routines in share/module_io_wrf.F– Many of the routines here are duplicative of
the routines in share/module_io_domain.F and an example of unnecessary layering in the WRF I/O software stack
– However, file does contain the base output_wrf and input_wrf routines in this file are what all the stream-specific wrappers (that are duplicated in the two layers)
![Page 62: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/62.jpg)
Field I/O
• Output_wrf and input_wrf– Contain hard coded WRF-specific meta-data puts (for
output) and gets (for input)• Whether meta-data is output or input is controlled by a flag in
the grid data structure• Meta data output is turned off when output_wrf is being
called as part of a "training write" within a two-stage open• It is turned on when it's called as part of an actual write
– Contain registry generated series of calls the WRF I/O API to write or read individual files
![Page 63: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/63.jpg)
Package-independent I/O API
• frame/module_io.F• These routines correspond to WRF I/O API specification• Start with the wrf_ prefix (package-specific routines start
with ext_package_)• The package-independent routines here contain logic for:
– selecting between formats (package-specific) based on the what stream is being written and what format is specified for that stream
– calling the external package as a parallel package (each process passes subdomain) or collecting and calling on a single WRF process
– passing the data off the the asynchronous quilt-servers instead of calling the I/O API from this task
![Page 64: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/64.jpg)
Package-specific I/O API
• Format specific implementations of I/O– external/io_netcdf/wrf_io.F90– external/io_int/io_int.F90– external/io_phdf5/wrf-phdf5.F90– external/io_mcel/io_mcel.F90
• The NetCDF version each contain a small program, diffwrf.F90, that uses the API read and then generate an ascii dump of a field that is readable by HMV (see: www.rotang.com) a small plotting program we use in-house for debugging and quick output.
• Diffwrf is also useful as a small example of how to use the I/O API to read a WRF data set
![Page 65: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/65.jpg)
Misc. Topics
![Page 66: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/66.jpg)
Misc. Topics
• Registry
• Error handling
• Time management
• Build mechanism
![Page 67: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/67.jpg)
Registry
• Overview of Registry program
• Survey of what is autogenerated
![Page 68: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/68.jpg)
Registry Source Files (in tools/)registry.c Main programreg_parse.c Parser Registry File and build AST
gen_allocs.c Generate allocate statementsgen_args.c Generate argument listsgen_comms.c Generate comms (STUBS or PACKAGE SPECIFIC)gen_config.c Generate namelist handling codegen_defs.c Generate variable/dummy arg declarationsgen_interp.c Generate nest interpolation codegen_mod_state_descr.c Generate frame/module_state_description.Fgen_model_data_ord.c Generate inc/model_data_ord.incgen_scalar_derefs.c Generate grid dereferencing code for non arraysgen_scalar_indices.c Generate code for 4D array indexinggen_wrf_io.c Generate calls to I/O API for fields
misc.c Utilities used in registry programmy_strtok.c " " " " "data.c Abstract syntax tree routinessym.c Symbol table (used by parser and AST)symtab_gen.c " " " " " "type.c Type handling, derived data types, misc.
![Page 69: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/69.jpg)
What the Registry Generates
• Include files in the inc directory…
![Page 70: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/70.jpg)
WRF Error Handling
• frame/module_wrf_error.F
• Routines for– Incremental debugging output WRF_DEBUG– Producing diagnostic messages
WRF_MESSAGE– Writing an error message and terminating
WRF_ERROR_FATAL
![Page 71: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/71.jpg)
WRF Time management
• Implementation of ESMF Time Manager
• Defined in external/esmf_time_f90
• Objects– Clocks– Alarms– Time Instances– Time Intervals
![Page 72: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/72.jpg)
WRF Time management
• Operations on ESMF time objects– For example: +, -, and other arithmetic is defined for
time intervals intervals and instances– I/O intervals are specified by setting alarms on clocks
that are stored for each domain; see share/set_timekeeping.F
– The I/O operations are called when these alarms "go off". see MED_BEFORE_SOLVE_IO in share/mediation_integrate.F
![Page 73: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/73.jpg)
WRF Build Mechanism
• Structure– Scripts:
• configure– Determines architecture using 'uname' then searchs the
arch/configure.defaults file for the list of possible compile options for that system. Typically the choices involved compiling for single-threaded, pure shared memory, pure distributed memory, or hybrid; may be other options too
– Creates the file configure.wrf, included by Makefiles
• compile [scenario]– Checks for existence of configure.wrf– Checks the environment for the core-specific settings such as
WRF_EM_CORE or WRF_NMM_CORE– Invokes the make command on the top-level Makefile passing it
information about specific targets to be built depending on the scenario argument to the script
• clean [-a]– Cleans the code, or really cleans the code
![Page 74: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/74.jpg)
WRF Build Mechanism
• Structure (continued)– arch/configure.defaults -- file containing settings for
various architectures– test directory
• Contains a set of subdirectories, each one for a different scenario. Includes idealized cases as well as directories for running real-data cases
• The compile script requires the name of one of these directories (for example "compile em_real") and based on that it compiles wrf.exe and the appropriate preprocessor (for example real.exe) and creates symbolic links from the test/em_real subdirectory to these executables
![Page 75: Monday January 26, 2004](https://reader035.vdocuments.mx/reader035/viewer/2022062221/568131cd550346895d9833dc/html5/thumbnails/75.jpg)
WRF Build Mechanism
• Structure (continued)– Top-level Makefile and Makefiles in subdirectories– The compile script invokes the top-level Makefile as:
"make scenario"– The top level Makefile, including rules and targets from the
configure.wrf file that was generated by the configure script, then recursively invokes Makefiles in subdirectories in order:
• external (external packages based on configure.wrf)• tools (builds registry)• frame (invokes registry and then builds framework)• shared (mediation layer and other modules and subroutines)• physics (physics package)• dyn_* (core specific code)• main (main routine and link to produce executables)