C-Coupler1: a Chinese community coupler for Earth

system modelingLi Liu, Cheng Zhang, Ruizhe Li, Guangwen Yang,

Bin Wang, Zhiyuan ZhangTsinghua University, Chinaliuli-cess@tsinghua.edu.cn

http://c-coupler.org/index.action1

Outline

• C-Coupler development

• C-Coupler1

• Bitwise identical reproducibility

• Future work

2

Milestones of C-Coupler development

3

Milestones TimeInitiation 2010.01

Main design and software design 2010.10

Prototype system of C-Coupler 2011.07

Common multi-dimensional remapping software CoR 2012.07

Prototype system of C-Coupler platform 2012.12

Early release of C-Coupler1 (beta version) and FGOALS-gc 2013.09

Parallel 3-D coupling 2013.10

Enhancement for bitwise identical reproducibility on the C-Coupler platform

2013.12

Release of C-Coupler1 2014.06

Target functions of C-Coupler Science

Flux computation 3-D coupling Two-way model nesting and interactive ensemble

Technology Modularization, extendibility High parallel efficiency

Application Powerful coupling functions, user friendliness Reliability, automatic error detection Reproducibility of simulation results

4

Architecture of models with C-Coupler

ATM ICE

LND OCN

Coupler component

C-Coupler

5

Interfaces

Interfaces

Process MGR Communication MGR

Grid MGR Remapping MGR

Restart MGR Decomposition MGR

Time MGR Data MGR

External coupling algorithms

Remapping algorithms

Scientific algorithms

I/O algorithms

C-Coupler runtime software system

Standardized component models

Component models code

ATMs OCNs LNDs ICEs Carbon …Component models configuration

Coupled models

configuration

Coupling flow configuration

Coupling generator

Runtime configuration

Coupling generator

Configuration system

Runtime software system

Software structure of C-Coupler

6

Software modulesStandardized component

models

External algorithms

Runtime configuration files ofexperiment models

C-Coupler

Create case From a default setting From an existing setting

Input data

ConfigureInitial or restart Output settings Start and stop time

Namelist Parallel settings Compiling options

Compile

Experimental setting package

Output data

Run case

Outline

• C-Coupler development

• C-Coupler1

• Bitwise identical reproducibility

• Future work

8

C-Coupler1• C-Coupler runtime software system for 3-D coupling

(in C++)– CoR1.0: A common multi-dimensional remapping

software for remapping, grid and field data management– Coupling interfaces (Fortran and C++)– Function managers, e.g., time manager, communication

manager, etc.– Parallelization

• C-Coupler platform: a runtime environment for model development, simulation and reproducibility

• Runtime configuration for the CPL6 coupling flow• Imported: CPL6 flux algorithms

9

Metadata for runtime configuration

• Timer– <unit of frequency, count of frequency, count of

delay>

• Field instance– <field name, component name, parallel

decomposition, grid name>

10

Metadata for runtime configuration

• External algorithm

11

Metadata for runtime configuration

• runtime algorithm list

• Runtime procedure

12

Data transfer and interpolation

• Data transfer– All fields (of different data types, on different grids, on

different parallel decompositions, or with different dimensions) to be transferred at the current time step can be packed into one message

• 3-D interpolation– 2-D+1-D implementation

• Spline is supported for 1-D interpolation– Offline and online– Parallel dynamic 3-D interpolation now (for example, for

coupling between AGCM and atmospheric chemistry model)

13

Example: parallel 2D flux coupling, global coupling, climate system model

GAMIL2 LICOM2

CLM3CICE4_LASG

C-Coupler1

14

FGOALS-gc

Coupler component in FGOALS-gc

15

Example: Sharing platform for GAMIL development

16

GAMILCLM2

FGOALS-gc

FGOALS-gc-colm

CLM3CoLM CLM4

GEOS-Chem

C-Coupler platform

…

WRF

MASNUM-Wav POM

C-Coupler

Example: parallel 3-D coupling, regional coupling, direct coupling

3-D coupling:Four choices for 1D

interpolation for vertical level

2D coupling

Direct coupling2D coupling 2D coupling

Example: model integration

18

• Integrating a standalone model versions, e.g., CESM1.2.1, CESM1.0.5, WRF and MOM4p1, onto the C-Coupler platform– Several configuration files– Less than 10 lines of source code in the main driver– Enhancement for bitwise-identical reproducibility to

the simulations

Outline

• C-Coupler development

• C-Coupler1

• Bitwise identical reproducibility

• Future work

19

Reproducibility

• A fundamental principle of scientific research

• More and more claims for reproducibility of published results– Nature family, Science and Geoscientific Model

Development, etc.

20

Bitwise identical reproducibility?

• It may be unnecessary, because climate simulations results are generally statistical characteristics of output data on time scales longer than a few months

• It was extremely difficult to achieve bitwise identical reproducibility– The whole simulation setting needs to be recorded

and recovered• Existing works show that climate simulation results

can be sensitive to round-off error

21

Experimental setups• Two fully coupled models: CESM1 and FGOALS-g2• CMIP5 historical experiments: 60 years (1850-1909)

22

ModelSimulation cases

#1 #2 #3 #4

CESM1 11.1_120_C1 11.1_128_C1 11.1_128_C2 12.1.3_128_C1

FGOALS-g2 11.1_104_C1 11.1_108_C1 11.1_108_C2 12.1.3_108_C1

Model LabelNumber of processes

ATM OCN LND ICE CPL GLC

CESM1120 120 120 120 120 120 120

128 128 128 128 128 128 128

FGOALS-g2104 30 18 24 20 12 -

108 30 18 24 20 16 -Model Label Compiling option

CESM1C1 -O2 -convert big_endian -assume byterecl -ftz -FR -fp-model precise

C2 -O2 -convert big_endian -assume byterecl -ftz -FR

FGOAL

S-g2

C1 -c -r8 -i4 -O2 -zero -132 -convert big_endian -assume byterecl -no-vec -mp1 -fp-model precise -fp-speculation=safe

C2 -c -r8 -i4 -O2 -zero -132 -convert big_endian -assume byterecl

Climatological mean TS by CESM1

23

Climatological mean TS by FGOALS-g2

24

Bitwise identical reproducibility is important to Earth system modeling

Current status of bitwise identical reproducibility of published results?

Design of a survey: 17 journals

25

Climate Dynamics Geophysical Research Letters

Geoscientific Model Development Global and Planetary Change

Global Biogeochemical Cycles Journal of Advances in Modelling Earth Systems

Journal of Climate Journal of Geophysical Research: Atmospheres

Journal of Hydrology Journal of Physical Oceanography

Journal of the Atmospheric Sciences Monthly Weather Review

Nature Nature Climate Change

Nature Geoscience Proceedings of the National Academy of Sciences of the United States of America

Quarterly Journal of the Royal Meteorological Society

Statistical characteristics of paper selection

26

Year of publishing 2006 2007 2008 2009 2010 2011 2012 2013 2014

Number of citations ≥10 ≥9 ≥8 ≥7 ≥5 ≥4 ≥3 ≥1 ≥0

Number of selected

papers35 35 42 41 42 45 48 46 17

Average number of

citations per paper92.1 74.5 65.9 52.6 26.0 31.8 20.4 5.1 0.4

Results of the survey

• No reply: 283 papers (80.6%)– No corresponding authors: 5 papers (1.4%)– Automatic email rejection: 66 papers (18.8%) – No active reply: 212 papers (60.4%)

• Replied without required information: 54 papers (15.4%)– Replied without required information and confirmation: 7 papers

(2%)– Inconvenient for reproduction: 47 papers (13.4%)

• Unsuccessful re-run: 4 papers (1.1%)• Successful re-run: 5 papers (1.4%)• Successful bitwise identical reproduction: 5 papers (1.4%)

27

Brief summary

• Fellow scientists heavily depend on the authors’ help to reproduce the published simulation results

• It is always inconvenient even impossible to recreate the same simulation setting as the whole simulation setting is rarely kept for a long time

• The authors still have to spend a lot of efforts to help the fellow scientists who want to reproduce these results, even when the whole simulation setting can be recalled

28

Bitwise identical reproducibility of Earth system modeling is currently at a very low level

29

Experimental setup package for technical reproducibility

GIT server and code version id for each component model

Reproducibility for model code

Code patch for each component model

SVN server and version id for each input data file

Reproducibility for input data

Check sum of each input data file

Reproducibility for input parametersScript for generating the input

parameter files of each component model

Reproducibility for parallel settings

Configuration file with the parallel settings of all component models

Reproducibility for Compiler and compiling options

Configuration file of compiling options of each component model

Information of compiler for each component model

Reproducibility for computer system

Name of the computer system

Log information for configuringUsername, computer name and

configuration time, and error and warning report for configuration

Log files for technical reproducibility

Log files for compiling

Log files for the execution of model simulation

Output files for technical reproducibilityName of model version

Description of model simulation

Time of the corresponding configuration

C-Coupler platform

Configure

Compile and run

Flowchart for achieving bitwise identical reproducibility on the C-Coupler platform

30

An example

31

Worldwide standard of bitwise identical reproducibility

• Any fellow scientists can independently obtain the whole simulation setting of published results and then can independently reproduce exactly the same simulation output

• Requires scientists’ actions, journals’ actions, model intercomparison projects’ actions, and technical supports.

32

A framework for achieving worldwide bitwise identical reproducibility

33

Outline

• C-Coupler development

• C-Coupler1

• Bitwise identical reproducibility

• Future work

34

Future work

• Coupling generator

• Parallel optimization

• Testing bed with benchmarks

• ASCII configuration file format XML format

• More coupling functions35

Thank you

36

Example: Computation performance

37

Time for a data transfer (low resolution)

Example: Computation performance

38Time for an interpolation (low resolution)