29 march 2007dkrz / m&d benutzerschulung, stephanie legutke1 session 1: agenda integrating model...

29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 1

Session 1: Agenda

Integrating Model and Data Infrastructure

Overview

History High level design

Source Code Management (SCM) Standard Compile Environment (SCE) Standard Running Environment (SRE)


IMDI: History

Develop for the European Earth System (ES) modelling community a software infrastructure to

compile run set up analyse coupled ES model experiments

and thereby establish a European ES research network

Since 2005: cooperation with PRISM Support Initiative (PSI)

see http://prism.enes.org continued development of SCE/SRE integration of data aspects (WDCC, data proc./vis.)

EU FP5 Project for Integrated Earth System Modeling (PRISM) Dec 2001-Nov 2004:


IMDI: High Level Design

Create a common infrastructure for ES modeling and data Modularity:

all components of IMDI independent Interoperability:

all components of IMDI work together Modularity in terms of experiment tasks Common look&feel for all (coupled) models and platforms Integrate best practices Modularity in terms of ES components


IMDI: Benefits

Facilitates exchange of component models

between modellersbetween institutes

communication between model users exchange of results intercomparison of simulations

Common look&feel: learn only once Choose what you like/need


SCM/SCE: Work Flow

Source & ScriptRepository

(SVN)

UpdateMakefiles

Modify source code

Retrieve source & script

code

Adapt source to SCE

CreateCompile Scripts

Compile

mo

del

.x


Use tools for SC analysis with[out] SCE list header file suffixes list directories containing header files list VPATH directoriesdetect multiple file namesmultiple MODULEs per file detectionMOULE name != file name detection

Use prerequisite files with[out] SCE Makefiles Use Makefile with[out] SCE compile script

SCM/SCE: Modular but Interoperable


SCE/SRE: Independent but Interoperable

Adapt I/O to SRE

Retrieve SRE code

Script codeRepository

(SVN)

CreateExp. Scripts

Configuresetup file

Launch Exp.

model.x

SCE


Post-processing

DB-Filling

LE-Visualisation

Preprocessing

Start Rerun Stop

www

FileArchive

FileArchive

HE-Visualisation

Processing

SRE-tasks: Modularity by Tasks

Archiving


Create Compilescripts

CompileCreate

Run Scripts

Run Experiment

user

SCE/SRE

model_tmpl.xsd model.xml tmpl.xsd

tmpl.xsd

model_tmpl.xsd

model_tmpl.xsd

model.xml model.xml

model.xml

SCE/SRE/DataManagement: Modular but Interoperable

model.xml


SCE / SRE: Modularity by ES Components

Applications:

PRISM (coupled) models (~12 components in 10 combinations on 10 platforms)

COSMOS model family of MPI-Met (cosmos-a, cosmos-as, cosmos-ao, cosmos-aob, cosmos-asob, cosmos-aso, cosmos-ob, cosmos-o)


ECHAM5ECHAM5

JSBACHJSBACH

MPIOM

Surface fluxes

Surface conditions

CO2

Grid

tra

nsfo

rmat

ion

Syn

chro

nisa

tion

OASIS3

HAMOCC

CO2

cosmos-acosmos-as cosmos-ocosmos-obcosmos-asob

29 March 2007 DKRZ / M&D Benutzerschulung, Stephanie Legutke 12cosmos-aocosmos-ao

ECHAM5ECHAM5 MPIOM

Surface fluxes

Surface conditions

Grid

tra

nsfo

rmat

ion

Syn

chro

nisa

tion

OASIS3

HAMOCC

CO2

JSBACHJSBACH

CO2

cosmos-aobcosmos-asob


Session 1: Agenda


Overview Source Code Management (SCM)

SVN repositorySource/script code directory structureBrowsing facility (being revised)Scripts to analyse and transform source code

Standard Compile Environment (SCE) Standard Running Environment (SRE)


SCM: Modularity by ES components & libraries

Entities: the OASIS3/4 coupler

interpolation and synchronisation component models

source code packages describing sub systems of the ES (e.g. sea ice, ocean, clouds, ...) main models:

components which create an executable (e.g. MPIOM) submodels:

component (optionally) linked to a main model(e.g. HAMOCC in MPIOM)

librariessource code package not representing subsystems of the ES


SCE: Libraries and Models

Components: Configurable => Can not be precompiled

OASIS3/4 Not configurable => Can be precompiled Not representing a subsystem of ES

Libraries: Not configurable => Can be precompiled Potentially usable by all components


s rc

su pp o rt

s rc

p sm ile

s rc

m o d -lib

lib

s rc

m od2

in c lu de

s rc s rc2

in c lu de

m od1

m od

s rc

SRCROO T

USE, CALL, INCLUDE

SVN/SCM: Source Code Directory src

ok

ok for call of SBR if mod2 is submodel of mod1


SCM: Restructure_directories.ksh

s rc1

in c lu de

s rc2

o c ean

s rc1

tra c e rs

s rc1 in c lu de

se a ice in c lu de file .F 90file .cf ile .h

m y m o d e l h is m o d e l h e r m o d e l

m od

s rc

ro o t

tra ce r_ s rc1 d irle ss file s

se a _ ice _ sc r1 se a _ ice_ in clu de

o cea n _s rc1 in c lu de

in c lu de

o cea n _s rc2

m y m o d e l h e r m o d e l h is m o d e l

m od

s rc

ro o t

tools/Restructure_directories.ksh


Source code conventions: no two files with same base name (file.f90 / file.c) one MODULE per file only MODULE name = file name

Use Coding_conventions.ksh for renaming / splitting in last two cases enables use of Create_prerequisites...

No change of source code proper required !

SCM: Coding_conventions.ksh


SCM/SCE/SRE: Implementation

Stand-alone models Unix-ksh (g)make m4, perl

Coupled models OASIS 3/4 (http://prism.enes.org) OASIS model interface library PSMILe; NetCDF library;

message passing library MPI-1/2; I/O library mpp_io=> use help function: script.ksh --help

=> get help from [email protected]


Session 1: Agenda


Overview Source Code Management (SCM) Standard Compile Environment (SCE)

SCE toolkit: Support for Makefile writing (templates) Creating compile scripts Maintaining Makefiles (prerequisites)

Standard Running Environment (SRE)


SCE/SRE: Directories

com pile

running

util

lib

m od

include

src tools

cplm od

data

SR C R O O T


SCE/SRE: Toolkit Directory util

header_xyz.h...

include

S itespecific_node.hO Sspecific_node.h...

include_node

C ppflags_m odel.h...

incl_m od_m odel

C om pile_m ode_m odel_node.h...

include_node_m odel

headers

C reate_C O M P_cpl_m odels.ksh

C reate_C O M P_libs.frm

C reate_C O M P_m odels.frm

C reate_prerequisites

C reate_prerequisites_cpp

tools

SC E_handbook

doc

com pile

C reate_TASK S.frm

tools

include

include_node

incl_m od_m odel

incl_cplm od_cplm od

include_node_cplm od

headers

SR E_handbook

doc adjunct_files

setup_cplm od_ID

setup functions

running

util


SCE: Toolkit Directory util/compile

header_xyz.h...

include

S itespecific_node.hO Sspecific_node.h...

include_node

C ppflags_m odel.h...

incl_m od_m odel

C om pile_m ode_m odel_node.h...

include_node_m odel

headers

C reate_C O M P_cpl_m odels.ksh

C reate_C O M P_libs.frm

C reate_C O M P_m odels.frm

C reate_prerequisites

C reate_prerequisites_cpp

tools

SC E_handbook

doc

com pile

util

SR C R O O T


No compile scripts are provided …

… but tools that create them …

… for the specific platform … for the specific component model or library … configured for the coupled combination of models

common look&feel

SCE: Compile Toolkit


Create_COMP_cpl_models.ksh:create all compile scripts for a coupled model (coupler, components, libraries)

Create_COMP_models.frm:create compile scripts for a component model or the coupler

Create_COMP_libs.frm:create compile scripts for the libraries

Create_prerequisitescreate a file containing USE or include based prerequisites

Create_prerequisites_cppcreate a prerequisite file controlling recompilation if cpp flags have changed

SCE: Compile Toolkit


SCE: Creating compile scripts

Create_COMP_cpl_models.ksh cplmod \ [--id=ID [-n=node [-s=MPI1 [-c=sun [--stdout=“-”]]]]

Create_COMP_models.frm mod1 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm mod2 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm mod3 -s=MPI1 -nnode –id=ID “mod1 mod2 mod3”Create_COMP_models.frm oasis3 -s=MPI1 -nnodeCreate_COMP_libs.frm -s=MPI1 -nnode

COMP_mod1_ID.kshCOMP_mod2_ID.kshCOMP_mod3_ID.kshCOMP_oasis3_MPI1.kshCOMP_libs.ksh

cplmod = mod1/mod2 & mod3 & oasis3:

mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].x oasis3.MPI[1,2].x

…. , psmile.MPI[1,2].a

mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].x oasis3.MPI[1,2].x

…. , psmile.MPI[1,2].a

mod1_mod2_ID.MPI[1,2].xmod2_ID.MPI[1,2].amod3_ID.MPI[1,2].xoasis3_MPI[1,2].x…. , psmile.MPI[1,2].a


SCE: Creating Compile Scripts

util/compile/headers/incl*/*.h

Compile scripts for - libraries (1 for all) - component mod

- coupler

m4

...include incl_mod_$model/GUIspecif_$model.hinclude include_$node/Sitespecific_$node.hinclude incl_mod_$model/Cppflags_$model.hinclude include_$node_$model/Compile_mode_$model_$node.hinclude Make_model.h

…

Create_COMP_models.frm model –nnode


cosmos-aob = echam5 & mpiom+hamocc & oasis3

Create_COMP_cpl_models.ksh cosmos-aob COMP_echam5_ID.ksh COMP_mpiom_ID.ksh default IDCOMP_hamocc_ID.ksh COMP_oasis3_MPI1.kshCOMP_libs.ksh

cosmos-a = echam5cosmos-as = echam5+jsbach cosmos-ao = echam5 & mpiom & oasis3cosmos-aob = echam5 & mpiom+hamocc & oasis3cosmos-asob = echam5+jsbach & mpiom+hamocc & oasis3cosmos-aso = echam5+jsbach & mpiom & oasis3cosmos-ob = mpiom+hamocccosmos-o = mpiom

SCE: Creating compile scripts (examples)

[--id=ds]


Creating compile scripts: examples

clm = clm Create_COMP_cpl_models.ksh clm

COMP_clm_ID.ksh

toyclim = toyatm & toyoce & toyche & oasis3Create_COMP_cpl_models.ksh toyclim

COMP_toyatm_ID.ksh COMP_toyoce_ID.ksh COMP_toyche_ID.ksh COMP_oasis3.kshCOMP_libs.ksh


Libraries

lib*.a

Libraries

lib*.aCompile libs of main

start

Compile sub model

Compilation: Main+Sub Model

Compile libs of sub

executable

Compilemain

model


mpiom_hamocc_ID[.MPIn].x

n = 1 or 2

COMP_hamocc_ID.ksh

COMP_libs.ksh support hamocc / libhamocc_ID[.MPIn].ampiom / libmpiom_hamocc _ID[.MPIn].a

hamocc / libhamocc_ID[.MPIn].ampiom / libmpiom_hamocc _ID[.MPIn].a

Local

libsupport.alibsupport.a

Central

start

CO

MP

_mp

iom

_ID

.ksh

Compilation: Main+Sub Model with central libraries

COMP_libs.ksh support


bin

m odn_ID

m od

libn

lib

build

lib

a rch

m od1

...

m odn

m od

lib1

...

libn

lib

src

SR C R O O T

*.o*.mod

*.a

modn*_ID.a

*_ID.x

*.o*.mod

SCE: Architecture Dependent Directories


Model compilation

*.o / *.mod*.o / *.mod

mod1_ID.xlib*.alib*.a

make

src directories:

make dir/Makefile

Top_Makefile

set / export par.create (in-doc.)

COMP_mod1_ID.ksh all / lib / clean / tar

library list:

make src / Makefile

*.o / *.mod*.o / *.mod

Top_Makefilemake

set / export par.create (in-doc.)

COMP_libs.ksh “library list”

with ‚all‘ only


All compilation is based on the (g)make software:Make ‘targets’ having ‘prerequisites’ with well defined ‘rules’ while avoiding redundant actions

Targets, rules & prerequisites are defined in file Makefile:

SCE: (GNU) Make

…target: prerequisite1, prerequisite2 ….

rules …

…‘what to make’: ‘what has to be made before’

‘how to make’ …


SCE Portable Makefiles

make [–f Makefile]

All non portable code is exported from the compile scripts

1 Makefile for each source directory with compiler input(*.F90, *.f90, *.F, *.f, *.c)

All Makefiles have the same look and feel

Prerequisites Tool for generation of *.o, *.h prerequisitesTool for generation of cpp flag prerequisites


SCE Make Targets

Executable: OASIS3_MPI1/2.x, ECHAM5_ID[_MPI1/2].x, *.x

Libraries: libblas.a, libsupport.a, libpsmile.MPI1/2.a, lib*.a

Model libraries: hamocc_ID[_MPI1/2].a, *_ID.a

Binary object code: *.o

Empty build directory

tar file of model code


SCE make prerequisites

Executable: Libraries and model libraries

echam5_ID.x: libsupport.a

mpiom_ID.x: libmpiom_ID.a

Library: object binarieslibsupport.a: *.o

Binary objects: F90 MODULEs and include files

file.o: module.mod (aliased to module.o)file.o: include.hfile.o: file.F90

Note:

with multiple target lines for the same target:

the join of all prerequisite sets is valid


SCE make rules $(PROG): $(LIBRARY)

$(F90) <flags> master.o echam5_ID.a support.a –o echam5_ID.x

$(LIBRARY): *.o$(AR) libsupport.a *.o

%.o: %.F90$(F90) <flags> -c $<

file.o: file.F90 $(F90) <special flags> -c $<

Note:

with multiple target lines for the same target:

the last rule is valid


Example Makefile: Targets & Rules

clean: rm -f $(LIBRARY) ls | xargs rm –f

lib: $(LIBRARY)all: $(PROG)

$(PROG): $(DEPLIBS) $(LIBRARY)$(F90) $(LDFLAGS) -o $@ $(MAINPRG).o $(LIBS)

$(LIBRARY): $(OBJS1) $(OBJS2) $(OBJS3) $(OBJS4) $(OBJS5)$(AR) $(ARFLAGS) $(LIBRARY) *.o

.SUFFIXES:

.SUFFIXES: .o .c .f .F .f90 .F90

%.o: %.F90$(F90) $(F90FLAGS) $(INCLS) -c $<

...

%.o: %.c $(CC) $(CCFLAGS) $(INCLSC) -c $<include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.deps-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.cpp

-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.special-include ../../../../src/mod/$(MODEL_DIR)$(strip $(SRC))/Make.inc.$(ARCH)

Create_prerequisites[_cpp]


Example Makefile: Definition of variables

SRCS1 = $(shell \find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) –name '*.F90‘ print)

OBJS1 = $(patsubst../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.F90, %.o, $(SRCS1))

...SRCS5 = $(shell \

find ../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)) -name '*.c' -print)OBJS5 = $(patsubst \

../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/%.c, %.o, $(SRCS5))VPATH = .:\

../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC)):\

../../../../src/mod/$(MODEL_DIR)/$(strip $(SRC))/include:\

../../../../src/mod/$(MODEL_DIR)/include:\

../../lib/psmile.${CHAN}:\$(NETCDF_INCLUDE):$(MPI_INCLUDE):$(SYS_INCLUDE):\../../../lib/$(MODEL_DIR):\../../../lib/$(SUBMOD1):\../../../lib:\../../../../src/mod/include/make_dummies

LIBRARY = ../../../lib/$(MODEL_DIR)/lib$(MODLIB).aPROG = ../../../bin/$(EXEC).xINCLS = -I../../../../src/mod/$(MODEL_DIR)/include \

-${I4mods}../../lib/clim.${CHAN}INCLSC = -I../../../../src/mod/$(MODEL_DIR)/include


Make.inc.deps

mod_unitncdf.o: mod_kinds_oasis.oalloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \

mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o

dealloc_src.o: mod_anais.o mod_analysis.o mod_coast.o mod_experiment.o \mod_extrapol.o mod_kinds_oasis.o mod_memory.o mod_nproc.o \mod_parallel.o mod_parameter.o mod_pipe.o mod_rainbow.o mod_sipc.o \mod_string.o mod_timestep.o mod_unitncdf.o

extrap.o: mod_extrapol.o mod_kinds_oasis.o mod_parameter.o mod_printing.o \mod_unit.o

getfld.o: mod_analysis.o mod_clim.o mod_experiment.o mod_hardware.o \mod_kinds_oasis.o mod_label.o mod_memory.o mod_parameter.o \mod_printing.o mod_sipc.o mod_string.o mod_timestep.o mod_unit.o \mod_unitncdf.o netcdf.inc

Create_prerequisites [model,library] directory


... mod_unitncdf.o: use_netCDF.cpp

alloc_src.o: use_comm_MPI1.cpp use_comm_MPI2.cpp use_comm_GSIP.cpp

dealloc_src.o: use_realtype_single.cpp ...

Make.inc.cpp

Create_prerequisites_cpp [model,library]


SCE Handbook: SVN:

http://svn-mad.zmaw.de/svn/mad/ \Model/IMDI/tags/’tag’/util/compile/doc

Web:http://mad.zmaw.de/imdi (bottom of page):

- pdf-version - html-version

Handbooks

29 march 2007dkrz / m&d benutzerschulung, stephanie legutke1 session 1: agenda integrating model...

Documents