technical details on how to use nicam chihiro kodama research institute for global change (rigc)...

Technical details on how to use NICAMTechnical details on how to use NICAM

Chihiro KodamaResearch Institute for Global Change (RIGC)

Japan Agency for Marine-Earth Science and Technology (JAMSTEC)

22Oct2009 COLA seminar

2

Contents of my talkContents of my talk

• How to use NICAM (technical details)– compile

– pre-processes (horizontal grid, external condition, initial condition)

– run (configuration)

– post-processes (lat-lon data)

– trouble shooting (runtime error)

• Following tutorial package (updated 2009.10.16) – http://nicam.jp/tutorial/ (password needed)

• Please feel free to ask questions!

3

Compile NICAMCompile NICAM

% ln –s Mkinclude.ES2 Mkinclude <- link Mkinclude.[arch] % emacs Mkinclude <- change $TOPPATH to NICAM-dir. % make

NOTE: Please mind that unit length of record must be 1 byte. In addition, input/output files are big-endian.

Intel ifort: -assume byterecl (compiler option) setenv F_UFMTENDIAN big

Earth Simulator 2: setenv F_RECLUNIT BYTE

TOPPATH: NICAM source code path

4

Managing Information Files (mnginfo)Managing Information Files (mnginfo)

• They contain:– number of regions

– linkages between one and other regions

– number of processes

– process-region mapping (Which process control a region?)

• mkmnginfo.cnf&MKMNGINFO rlevel = 5, prc_num = 320, output_fname = ‘rl5-prc320.info’,/

% ${TOPPATH}/bin/mkmnginfo

necessary to run MPI-based program with 320 processes, rlevel=5.

Concept of rlevel will be explained later.

5

Concept of Horizontal GridConcept of Horizontal Grid• What is “glevel” ?

– Total Grid Division Level

Figures from http://www.ccsr.u-tokyo.ac.jp/~satoh/nicam/ico.html

g: glevelnumber of rhomboses = 10×4g

S: Earth Surface [m2]resolution ~ sqrt [S / (10×4g)]

glevel 07141 km

glevel 13570 km

glevel 21785 km

glevel 3893 km

glevel 4446 km

glevel 5223 km

glevel resolution 6 112 km 7 56 km 8 28 km 9 14 km 10 7 km 11 3.5 km

6

Concept of Horizontal GridConcept of Horizontal Grid• What is “rlevel” ?

– Region Division Level– Designed for MPI program

– File name: var-name.rgn????? e.g. sst.rgn00000 ~ sst.rgn10239– Note: As rlevel increases, total memory consumption is increased, especially when

glevel-rlevel is small. • due to overlapped grid along edge of the region.

rlevel 010 regions / globe

rlevel # of regions 0 10 1 40 2 160 3 640 4 2560 5 10240

If using 20 processes,128 regions per process

rlevel 140 regions / globe

regionregion

7

Flow of Horizontal Grid GenerationFlow of Horizontal Grid Generation

mkgrid

mkgcgrid

splitgrid

_grid.rgn????? (for rlevel=0)

_grid.rgn????? (for rlevel > 0)

Horizontal Grid Filesgrid.rgn?????

provided in the package

increase rlevel

modify grid (gravitational-centered)

if rlevel=0

generate grid

8

Generating Horizontal Grid (1/2) splitgridGenerating Horizontal Grid (1/2) splitgrid

• splitgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 0, vlayer = 1, rgnmngfname ='../../mnginfo/data/rl00-prc05.info', /&SPLITGRIDPARAM input_base = '../../../NICAM_DATABASE/_hgrid/gl10/rl00/grid', output_base = 'data/_grid', alt_rlevel = 5,/

% mpirun -np 5 ${TOPPATH}/bin/splitgrid

Output: _grid.rgn00000, _grid.rgn00001, ..., _grid.rgn10239

9

Generating Horizontal Grid (2/2) mkgcgridGenerating Horizontal Grid (2/2) mkgcgrid

• mkgcgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&MKGCGRIDPARAM input_base = '../splitgrid/data/_grid', output_base = 'data/grid',/

% mpirun -np 5 ${TOPPATH}/bin/mkgcgrid

Output: grid.rgn00000, grid.rgn00001, ..., grid.rgn10239

10

External Condition Files (external_data/)External Condition Files (external_data/)• MATSIRO/

– topog: topography

– topog_std: standard deviation of topography

– albedo_*: surface albedo

– gradz: tangent of mean surface slope

– lai: leaf area index

– slidx: soil index

– veget_mat: vegetation index

• O3/ : ozone• OCN/ (for nudging slab ocean)

– sst: sea surface temperature [K]

– ice: sea ice mass [kg/m2]• ice is NEITHER sea ice concentration NOR sea ice fraction.

MATSIRO : Minimal Advanced Treatments of Surface Interaction and RunOff

11

Topography Files (MATSIRO/)Topography Files (MATSIRO/)

• To avoid numerical instability, use topography files included in the tutorial package.– For higher rlevel run, split the above files using cnvrlev,

which increases rlevel of general data

– cnvrlev.cnf&ADMPARAM ... /&SPLITGRIDPARAM num_data = 1, input_base = '../../NICAM_DATABASE/sfcdata/gl10/rl04/topog', input_direct = .false., output_base = 'data/topog', output_direct = .false., output_rlevel = 5,/

% mpirun -np 5 ${TOPPATH}/bin/cnvrlev

Output: topog.rgn00000, topog.rgn00001, ..., topog.rgn10239

12

Other MATSIRO Files / O3 FilesOther MATSIRO Files / O3 Files

• See tutorial package for details. Probably, all you have to do is modify glevel & rlevel.– mklanddata2.cnf

– mko3.cnf

13

OCN Files (1/2)OCN Files (1/2)

• SST: sea surface temperature [K]• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../dataset/oisst_v2/sst.grd', output_sst_base = './data/sst', num_data = 8, data_periodic = .false., imax = 360, jmax = 180, slat = 89.5, slon = 0.5, dlat = 1.0, dlon = 1.0, opt_southward = .false., direct_access = .true.,/

% mpirun -np 5 ${TOPPATH}/bin/mkgisst

14

OCN Files (2/2)OCN Files (2/2)

• ICE: sea ice mass [kg/m2]– In our previous package, sea ice fraction is used. It’s

wrong!

• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../NICAM_DATABASE/rawdata/hadsst1979-1999_clm.dat', ice_lldata_fname = '../../../NICAM_DATABASE/rawdata/cmip3sit1979-1999_clm.dat', output_sst_base = './data/hadsst1979-1999_clm', output_ice_base = './data/cmip3ice1979-1999_clm', num_data = 12, data_periodic = .false., direct_access = .true., offset = 0.0, ice_fact = 1000.0,/

NOTE: SST above (HadSST) is used only for modifying ice.

15

Initial Condition Files (1/2)Initial Condition Files (1/2)

• Atmosphere– 3D velocity– temperature– water vapor mixing ratio– cloud water mixing ratio

• Ocean (for slab ocean)– SST– sea ice fraction & mass & surface temperature– snow on sea ice

• Land (for MATSIRO)– soil moisture & temperature– surface accumulated snow

Most variables can be obtained from NCEP Final Analysis

16

Initial Condition Files (2/2)Initial Condition Files (2/2)• Atmosphere

– mkinit_atm.cnf&MKINIT_ATM_PARAM glevel = 10, rlevel = 5, rain_type = 'COLD', plus_tke = .true., hgrid_basename = '../../../NICAM_DATABASE/hgrid/gl10/rl05/grid.', vgrid_filename = '../../grid/vgrid/data/vgrid40.dat', topo_basename = '../../../NICAM_DATABASE/sfcdata/gl10/rl05/topog', ncep_atm_filename = '../atm.dat', init_atm_basename = 'init_atm',/

After you prepare atm.dat (see tutorial for data format),

% ${TOPPATH}/bin/mkinit_atm

Preparing ocean and land conditions is similar.

17

Prep. for Icosahedral Data Prep. for Icosahedral Data Lat-Lon Data Lat-Lon Data• Lat-Lon mapping files (llmap) are necessary

– mkllmap.cnf&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&GRDPARAM hgrid_fname = '../../../NICAM_DATABASE/hgrid/gl10/rl04/grid‘,/&LATLONPARAM imax = 5120, jmax = 2560,/&MKLLMAP_PARAM output_dir = './data/‘,/

% mpirun -np 5 ${TOPPATH}/bin/mkllmap

Output: llmap.rgn00000, llmap.rgn00001, ..., llmap.rgn10239

18

Final Step for NICAM simulationFinal Step for NICAM simulation

• You have to prepare nhm_driver.cnf.• See tutorial package for more detailed descriptions of

nhm_driver.cnf

% mpirun -np 320 ${TOPPATH}/bin/nhm_driver

19

nhm_driver.cnf (Run Type Settings)nhm_driver.cnf (Run Type Settings)&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'COLD', NDIFF_LOCATION = 'IN_LARGE_STEP2' RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'NSW6', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'NONE', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 1, SFC_DIV_NUM = 1,/

For high resolution run For low resolution run

&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'CLOUD_PARAM', NDIFF_LOCATION = 'IN_LARGE_STEP2', RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'LSC', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'PAS', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 24, SFC_DIV_NUM = 12,/

20

nhm_driver.cnf (Time Step Control)nhm_driver.cnf (Time Step Control)&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.

start data/time

Δt [s]

number of time step

21

nhm_driver.cnf (Initial Conditions 1/2)nhm_driver.cnf (Initial Conditions 1/2)&RESTARTPARAM input_basename = '../../initial_data/ATM/init_atm' input_direct_access = .true., output_direct_access = .true., trc_vmax_input = 3,/##### <--- when restart, delete “TRC_VMAX_INPUT=3”,&NM_OCEAN_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/OCN/init_ocn', merged_output_restfname = 'restart_ocn',/&NM_LAND_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/LND/init_lnd', merged_output_restfname = 'restart_lnd', opt_albsfc_in = .false., opt_albsfc_out = .true.,/###### <--- When restart, opt_albsfc_in = .true.

Atmosphere

Ocean

Land

22

nhm_driver.cnf (Initial Conditions 2/2)nhm_driver.cnf (Initial Conditions 2/2)

• When you restart run,– &RESTARTPARAM

• change input_basename• delete trc_vmax_input

– &NM_OCEAN_INIT• change merged_input_restfname

– &NM_LAND_INIT• change merged_input_restfname• let opt_albsfc_in = .true.

– &DIAGVARPARAM• set tb_fname

&DIAGVARPARAM tb_fname = 'NONE', input_direct_access = .true., output_direct_access = .true., /###### <--- When restart, set TB_FNAME (restart_diagvar)

Normally, no initial conditions for turbulence

Turbulence

23

nhm_driver.cnf (External Conditions 1/2)nhm_driver.cnf (External Conditions 1/2)• Case 1: Time invariant field

– e.g. ‘SLIDX’, ‘GRTANS’, ‘GRALB’, ‘GRALBN’

• Case 2: Monthly-constant field– e.g. ‘OZONE’

&NM_EXTDATA dataname ='OZONE', fname = '../../external_data/O3/data/amip2o3', num_of_data = 12, opt_monthly_cnst = .true., layer_type = 'ATM',/

&NM_EXTDATA dataname = 'GRTANS', fname = '../../external_data/MATSIRO/data/gradz/gradz', num_of_data = 1, layer_type = 'SFC',/

24

nhm_driver.cnf (External Conditions 2/2)nhm_driver.cnf (External Conditions 2/2)• Case 3: Step-by-step field (with time interpolation)

– e.g. ‘SST’, ‘ICE’, ‘GRLAI’

&NM_EXTDATA dataname = 'SST', fname = '../../external_data/OCN/data/sst', num_of_data = 8, data_date(:,1) = 2004, 5,26,0,0,0, data_date(:,2) = 2004, 6,02,0,0,0, data_date(:,3) = 2004, 6,09,0,0,0, data_date(:,4) = 2004, 6,16,0,0,0, data_date(:,5) = 2004, 6,23,0,0,0, data_date(:,6) = 2004, 6,30,0,0,0, data_date(:,7) = 2004, 7,07,0,0,0, data_date(:,8) = 2004, 7,14,0,0,0, layer_type = 'SFC', opt_periodic_year = .false.,/###### <--- PLEASE CHECK THE DATES BEFORE RUNNING!!!

25

nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)

• ktype = ‘3D’, ‘2D’, ‘GO’ (ocean), ‘GL’(land), ‘ISCCP’

&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360,/&NMHIST ktype = ‘2D’, item = ‘sl_slp’, file = ‘sa_slp’, output_type = ‘AVERAGE’, step = 120,/

variable namefile name ( ms_tem.rgn?????)

‘SNAPSHOT’or

‘AVERAGE’

timestep interval of outputif DTL=30s, step = 120: 1 hourly step = 360: 3 hourly

26

nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)

• &NMHISD: default setting of &NMHIST

• Output with pressure levels

&NMHISD direct_access = .TRUE., no_vintrpl = .false.,/

.true. for z* coordinate

.false. for z coordinate

&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360, out_prelev = .true.,/

&NMHISD... npreslev = 23,   pres_levs = 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 10, 7, 5, 3, 2, 1, 0.4,/

27

Run NICAM !Run NICAM !

• See tutorial package for more detailed descriptions of nhm_driver.cnf

% mpirun -np 320 ${TOPPATH}/bin/nhm_driver

Output: ms_tem.rgn?????, sa_slp.rgn?????, .....(huge number of files are created)

28

Icosahedral Data Icosahedral Data Lat-Lon Data Lat-Lon Data

• ico2ll.cnf&ICO2LL_PARAM glevel = 10, rlevel = 5, input_dir = './icodata/', output_dir = './lldata/' info_fname = 'history.info', llmap_dir = '../../grid/llmap/data/' llmap_base = 'llmap', lon_swap = .true., input_size = 4, init_year = 2004, init_month = 6, init_day = 1,/

% ${TOPPATH}/bin/ico2ll

automatically generated by nhm_driver

Output: ms_tem.ctl, ms_tem.grd, sa_slp.ctl, sa_slp.grd, .....

29

Restarting RunRestarting Run

• Followings should be changed in nhm_driver.cnf:– &TIMEPARAM: start_*

– External condition• &RESTARTPARAM

• &NM_OCEAN_INIT

• &NM_LANDINIT

• &DIAGVARPARAM

30

Changing GlevelChanging Glevel

• Followings should be changed in nhm_driver.cnf:– &RUNCONFPARAM (e.g. cloud resolving / cloud param.)

– &ADMPARAM• glevel, rlevel, rgnmngfname

– &NUMFILTERPARAM (as glevel increase by 1,)• Kh_coef_maxlim , Kh_coef_minlim, alpha_d : divided by 8

• gamma_h_lap1 : divided by 2

– &TIMEPARAM• dtl, lstep_max (if necessary)

– &NM_RD_TIME• TINTV= 10 mins for gl9, but 5 mins for gl10 and 11

– File names

31

Handling Runtime ErrorHandling Runtime Error

• Model bug? Configuration mistake? Numerical instability?• Model run diagnosis

– msg.????? : message from NICAM– standard out/err

• Check List– Where is the run fail? What module/subroutine/line?– Are the prescribed files such as external_data and initial_data correct?

• check them using ico2ll• unit of variables• glevel/rlevel• file name, little/big endian

– compiler option (such as “-assume byterecl” in intel ifort)– memory enough for the run?– parameters in nhm_driver.cnf correct?– Does the error also occur in the lower glevel/rlevel?

If the runtime error seems to occur due to numerical instability, ...

32

Numerical Filter in nhm_driver.cnfNumerical Filter in nhm_driver.cnf&NUMFILTERPARAM dep_hgrid = .false. DEEP_EFFECT = .false., hdiff_fact_rho = 0.01D0, hdiff_fact_q = 0.0d0,

hdiff_type = 'NONLINEAR1', cfact = 2.0D0, Kh_coef_maxlim = 3.125D+12, Kh_coef_minlim = 3.125D+11, ZD_hdiff_nl = 20000D0,

gamma_h_lap1 = 6.25D+5, ZD_hdiff_lap1 = 20000.0D0,

divdamp_type = 'DIRECT', lap_order_divdamp = 2, alpha_d = 3.125D+11, alpha_dv = 0.00D0,

gamma_v = 0.0D0,

ZD = 25000.0d0, alpha_r = 0.0D0,/

horizontal diffusion

Laplacian-type horizontal diffusion

3D divergence damping

vertical diffusion

Rayleigh friction

In addition, 2D divergence damping

33

Other Configurations in nhm_driver.cnfOther Configurations in nhm_driver.cnf

&nm_tb_my2moist ICE = .true., LENGTH_TYPE = '3KM', NU_MIN = 0.1D0, KH_MIN = 0.1D0, N2max = 0.0016, /##### <--- If unstable, reduce the N2max.

&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.

34

Adjusting Initial ConditionsAdjusting Initial Conditions

• Tool from Tatsuya Seiki (CCSR, Univ. Tokyo)– modify pressure to achieve hydrostatic balance

– modify vertical velocity using continuity equation

35

SummarySummary

• Please inform us if you find bugs, misleading points, suggestions, etc in NICAM tutorial package.

Thank you

36

37

##### Appendix / Memo ########## Appendix / Memo #####

38

0. Who am I ?0. Who am I ?

• Research interest– storm track

– stratosphere dynamics

– global warming

– role of cloud in climate system

• Misc– GrADS script library

39

Memory Consumption as rlevel increaseMemory Consumption as rlevel increase

40

Generating Vertical GridGenerating Vertical Grid

• mkvlayer.cnf&MKVLAYER_CNF num_of_layer = 40, layer_type = 'GIVEN', infname = './vgrid40.txt', outfname = './data/vgrid40.dat',/

0.000000000000000E+000161.683000000000335.958000000000523.806000000000726.285000000000944.5340000000001179.781000000001433.349000000001706.667000000002001.272000000002318.822000000002661.104000000003030.045000000003427.720000000003856.368000000004318.400000000004816.418000000005353.223000000005931.837000000006555.516000000007227.769000000007952.380000000008733.428000000009575.3060000000010482.753000000011460.876000000012515.179000000013651.596000000014876.521000000016196.850000000017620.010000000019154.012000000020807.488000000022589.744000000024510.811000000026581.496000000028813.457000000031219.252000000033812.418000000036607.555000000039620.3830000000

TODO: how to create refstateorNo explanation of vertical grid

41

refstaterefstate

42

nhm_driver.cnfnhm_driver.cnf&nm_rd_time

TINTV=10.0D0,TUNIT='MIN'

/###### <--- TINTV = 10min for gl9, but 5min for gl10 and 11.

43

nhm_driver.cnf (Resolution, Grid)nhm_driver.cnf (Resolution, Grid)

&ADMPARAM glevel = 10, rlevel = 5, vlayer = 40, rgnmngfname = '../../mnginfo/data/rl05-prc320.info',/&GRDPARAM hgrid_fname = '../../grid/hgrid/data/grid', vgrid_fname = '../../grid/vgrid/data/vgrid40.dat', topo_fname = '../../external_data/MATSIRO/data/topog/topog', toposd_fname = '../../external_data/MATSIRO/data/topog_std/topog_std', vegeindex_fname = '../../external_data/MATSIRO/data/veget_mat/veget_mat'/

44

nhm_driver.cnf (Ocean)nhm_driver.cnf (Ocean) &nm_ocean_mixedlayer osstfx = .false.,

oicefx = .false.,oicrfx = .false.,TNUGST = 432000.0D0,TNUGSI = 432000.0D0,

/###### <--- The relaxation times of SST and Sea ICE are 5 days.