technical details on how to use nicam chihiro kodama research institute for global change (rigc)...
TRANSCRIPT
Technical details on how to use NICAMTechnical details on how to use NICAM
Chihiro KodamaResearch Institute for Global Change (RIGC)
Japan Agency for Marine-Earth Science and Technology (JAMSTEC)
22Oct2009 COLA seminar
2
Contents of my talkContents of my talk
• How to use NICAM (technical details)– compile
– pre-processes (horizontal grid, external condition, initial condition)
– run (configuration)
– post-processes (lat-lon data)
– trouble shooting (runtime error)
• Following tutorial package (updated 2009.10.16) – http://nicam.jp/tutorial/ (password needed)
• Please feel free to ask questions!
3
Compile NICAMCompile NICAM
% ln –s Mkinclude.ES2 Mkinclude <- link Mkinclude.[arch] % emacs Mkinclude <- change $TOPPATH to NICAM-dir. % make
NOTE: Please mind that unit length of record must be 1 byte. In addition, input/output files are big-endian.
Intel ifort: -assume byterecl (compiler option) setenv F_UFMTENDIAN big
Earth Simulator 2: setenv F_RECLUNIT BYTE
TOPPATH: NICAM source code path
4
Managing Information Files (mnginfo)Managing Information Files (mnginfo)
• They contain:– number of regions
– linkages between one and other regions
– number of processes
– process-region mapping (Which process control a region?)
• mkmnginfo.cnf&MKMNGINFO rlevel = 5, prc_num = 320, output_fname = ‘rl5-prc320.info’,/
% ${TOPPATH}/bin/mkmnginfo
necessary to run MPI-based program with 320 processes, rlevel=5.
Concept of rlevel will be explained later.
5
Concept of Horizontal GridConcept of Horizontal Grid• What is “glevel” ?
– Total Grid Division Level
Figures from http://www.ccsr.u-tokyo.ac.jp/~satoh/nicam/ico.html
g: glevelnumber of rhomboses = 10×4g
S: Earth Surface [m2]resolution ~ sqrt [S / (10×4g)]
glevel 07141 km
glevel 13570 km
glevel 21785 km
glevel 3893 km
glevel 4446 km
glevel 5223 km
glevel resolution 6 112 km 7 56 km 8 28 km 9 14 km 10 7 km 11 3.5 km
6
Concept of Horizontal GridConcept of Horizontal Grid• What is “rlevel” ?
– Region Division Level– Designed for MPI program
– File name: var-name.rgn????? e.g. sst.rgn00000 ~ sst.rgn10239– Note: As rlevel increases, total memory consumption is increased, especially when
glevel-rlevel is small. • due to overlapped grid along edge of the region.
rlevel 010 regions / globe
rlevel # of regions 0 10 1 40 2 160 3 640 4 2560 5 10240
If using 20 processes,128 regions per process
rlevel 140 regions / globe
regionregion
7
Flow of Horizontal Grid GenerationFlow of Horizontal Grid Generation
mkgrid
mkgcgrid
splitgrid
_grid.rgn????? (for rlevel=0)
_grid.rgn????? (for rlevel > 0)
Horizontal Grid Filesgrid.rgn?????
provided in the package
increase rlevel
modify grid (gravitational-centered)
if rlevel=0
generate grid
8
Generating Horizontal Grid (1/2) splitgridGenerating Horizontal Grid (1/2) splitgrid
• splitgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 0, vlayer = 1, rgnmngfname ='../../mnginfo/data/rl00-prc05.info', /&SPLITGRIDPARAM input_base = '../../../NICAM_DATABASE/_hgrid/gl10/rl00/grid', output_base = 'data/_grid', alt_rlevel = 5,/
% mpirun -np 5 ${TOPPATH}/bin/splitgrid
Output: _grid.rgn00000, _grid.rgn00001, ..., _grid.rgn10239
9
Generating Horizontal Grid (2/2) mkgcgridGenerating Horizontal Grid (2/2) mkgcgrid
• mkgcgrid.cnf (e.g. glevel=10, rlevel=5)&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&MKGCGRIDPARAM input_base = '../splitgrid/data/_grid', output_base = 'data/grid',/
% mpirun -np 5 ${TOPPATH}/bin/mkgcgrid
Output: grid.rgn00000, grid.rgn00001, ..., grid.rgn10239
10
External Condition Files (external_data/)External Condition Files (external_data/)• MATSIRO/
– topog: topography
– topog_std: standard deviation of topography
– albedo_*: surface albedo
– gradz: tangent of mean surface slope
– lai: leaf area index
– slidx: soil index
– veget_mat: vegetation index
• O3/ : ozone• OCN/ (for nudging slab ocean)
– sst: sea surface temperature [K]
– ice: sea ice mass [kg/m2]• ice is NEITHER sea ice concentration NOR sea ice fraction.
MATSIRO : Minimal Advanced Treatments of Surface Interaction and RunOff
11
Topography Files (MATSIRO/)Topography Files (MATSIRO/)
• To avoid numerical instability, use topography files included in the tutorial package.– For higher rlevel run, split the above files using cnvrlev,
which increases rlevel of general data
– cnvrlev.cnf&ADMPARAM ... /&SPLITGRIDPARAM num_data = 1, input_base = '../../NICAM_DATABASE/sfcdata/gl10/rl04/topog', input_direct = .false., output_base = 'data/topog', output_direct = .false., output_rlevel = 5,/
% mpirun -np 5 ${TOPPATH}/bin/cnvrlev
Output: topog.rgn00000, topog.rgn00001, ..., topog.rgn10239
12
Other MATSIRO Files / O3 FilesOther MATSIRO Files / O3 Files
• See tutorial package for details. Probably, all you have to do is modify glevel & rlevel.– mklanddata2.cnf
– mko3.cnf
13
OCN Files (1/2)OCN Files (1/2)
• SST: sea surface temperature [K]• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../dataset/oisst_v2/sst.grd', output_sst_base = './data/sst', num_data = 8, data_periodic = .false., imax = 360, jmax = 180, slat = 89.5, slon = 0.5, dlat = 1.0, dlon = 1.0, opt_southward = .false., direct_access = .true.,/
% mpirun -np 5 ${TOPPATH}/bin/mkgisst
14
OCN Files (2/2)OCN Files (2/2)
• ICE: sea ice mass [kg/m2]– In our previous package, sea ice fraction is used. It’s
wrong!
• mkgisst.cnf&ADMPARAM ... /&GRDPARAM ... /&MKGISSTPARAM sst_lldata_fname = '../../../NICAM_DATABASE/rawdata/hadsst1979-1999_clm.dat', ice_lldata_fname = '../../../NICAM_DATABASE/rawdata/cmip3sit1979-1999_clm.dat', output_sst_base = './data/hadsst1979-1999_clm', output_ice_base = './data/cmip3ice1979-1999_clm', num_data = 12, data_periodic = .false., direct_access = .true., offset = 0.0, ice_fact = 1000.0,/
NOTE: SST above (HadSST) is used only for modifying ice.
15
Initial Condition Files (1/2)Initial Condition Files (1/2)
• Atmosphere– 3D velocity– temperature– water vapor mixing ratio– cloud water mixing ratio
• Ocean (for slab ocean)– SST– sea ice fraction & mass & surface temperature– snow on sea ice
• Land (for MATSIRO)– soil moisture & temperature– surface accumulated snow
Most variables can be obtained from NCEP Final Analysis
16
Initial Condition Files (2/2)Initial Condition Files (2/2)• Atmosphere
– mkinit_atm.cnf&MKINIT_ATM_PARAM glevel = 10, rlevel = 5, rain_type = 'COLD', plus_tke = .true., hgrid_basename = '../../../NICAM_DATABASE/hgrid/gl10/rl05/grid.', vgrid_filename = '../../grid/vgrid/data/vgrid40.dat', topo_basename = '../../../NICAM_DATABASE/sfcdata/gl10/rl05/topog', ncep_atm_filename = '../atm.dat', init_atm_basename = 'init_atm',/
After you prepare atm.dat (see tutorial for data format),
% ${TOPPATH}/bin/mkinit_atm
Preparing ocean and land conditions is similar.
17
Prep. for Icosahedral Data Prep. for Icosahedral Data Lat-Lon Data Lat-Lon Data• Lat-Lon mapping files (llmap) are necessary
– mkllmap.cnf&ADMPARAM glevel = 10, rlevel = 5, vlayer = 1, rgnmngfname = '../../mnginfo/data/rl05-prc05.info',/&GRDPARAM hgrid_fname = '../../../NICAM_DATABASE/hgrid/gl10/rl04/grid‘,/&LATLONPARAM imax = 5120, jmax = 2560,/&MKLLMAP_PARAM output_dir = './data/‘,/
% mpirun -np 5 ${TOPPATH}/bin/mkllmap
Output: llmap.rgn00000, llmap.rgn00001, ..., llmap.rgn10239
18
Final Step for NICAM simulationFinal Step for NICAM simulation
• You have to prepare nhm_driver.cnf.• See tutorial package for more detailed descriptions of
nhm_driver.cnf
% mpirun -np 320 ${TOPPATH}/bin/nhm_driver
19
nhm_driver.cnf (Run Type Settings)nhm_driver.cnf (Run Type Settings)&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'COLD', NDIFF_LOCATION = 'IN_LARGE_STEP2' RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'NSW6', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'NONE', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 1, SFC_DIV_NUM = 1,/
For high resolution run For low resolution run
&RUNCONFPARAM EIN_TYPE = 'SIMPLE', RAIN_TYPE = 'CLOUD_PARAM', NDIFF_LOCATION = 'IN_LARGE_STEP2', RAD_TYPE = 'ISCCP', TRC_ADV_TYPE = 'MIURA2004', TRC_NEG_FIX = 'OFF', AF_TYPE = 'NONE', MP_TYPE = 'LSC', RD_TYPE = 'MSTRNX_AR5', ND_TYPE = 'NONE', FR_TYPE = 'NONE', SV_TYPE_T = 'GIVEN', SV_TYPE_QV = 'SATURATION', SF_TYPE = 'LOUIS', TB_TYPE = 'MY2MOIST', CP_TYPE = 'PAS', GWD_TYPE = 'NONE', LAND_TYPE = 'MATSIRO', OCEAN_TYPE = 'MIXEDLAYER', TB_DIV_NUM = 24, SFC_DIV_NUM = 12,/
20
nhm_driver.cnf (Time Step Control)nhm_driver.cnf (Time Step Control)&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.
start data/time
Δt [s]
number of time step
21
nhm_driver.cnf (Initial Conditions 1/2)nhm_driver.cnf (Initial Conditions 1/2)&RESTARTPARAM input_basename = '../../initial_data/ATM/init_atm' input_direct_access = .true., output_direct_access = .true., trc_vmax_input = 3,/##### <--- when restart, delete “TRC_VMAX_INPUT=3”,&NM_OCEAN_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/OCN/init_ocn', merged_output_restfname = 'restart_ocn',/&NM_LAND_INIT merged_rst_in = .true., merged_rst_out = .true., merged_input_restfname = '../../initial_data/LND/init_lnd', merged_output_restfname = 'restart_lnd', opt_albsfc_in = .false., opt_albsfc_out = .true.,/###### <--- When restart, opt_albsfc_in = .true.
Atmosphere
Ocean
Land
22
nhm_driver.cnf (Initial Conditions 2/2)nhm_driver.cnf (Initial Conditions 2/2)
• When you restart run,– &RESTARTPARAM
• change input_basename• delete trc_vmax_input
– &NM_OCEAN_INIT• change merged_input_restfname
– &NM_LAND_INIT• change merged_input_restfname• let opt_albsfc_in = .true.
– &DIAGVARPARAM• set tb_fname
&DIAGVARPARAM tb_fname = 'NONE', input_direct_access = .true., output_direct_access = .true., /###### <--- When restart, set TB_FNAME (restart_diagvar)
Normally, no initial conditions for turbulence
Turbulence
23
nhm_driver.cnf (External Conditions 1/2)nhm_driver.cnf (External Conditions 1/2)• Case 1: Time invariant field
– e.g. ‘SLIDX’, ‘GRTANS’, ‘GRALB’, ‘GRALBN’
• Case 2: Monthly-constant field– e.g. ‘OZONE’
&NM_EXTDATA dataname ='OZONE', fname = '../../external_data/O3/data/amip2o3', num_of_data = 12, opt_monthly_cnst = .true., layer_type = 'ATM',/
&NM_EXTDATA dataname = 'GRTANS', fname = '../../external_data/MATSIRO/data/gradz/gradz', num_of_data = 1, layer_type = 'SFC',/
24
nhm_driver.cnf (External Conditions 2/2)nhm_driver.cnf (External Conditions 2/2)• Case 3: Step-by-step field (with time interpolation)
– e.g. ‘SST’, ‘ICE’, ‘GRLAI’
&NM_EXTDATA dataname = 'SST', fname = '../../external_data/OCN/data/sst', num_of_data = 8, data_date(:,1) = 2004, 5,26,0,0,0, data_date(:,2) = 2004, 6,02,0,0,0, data_date(:,3) = 2004, 6,09,0,0,0, data_date(:,4) = 2004, 6,16,0,0,0, data_date(:,5) = 2004, 6,23,0,0,0, data_date(:,6) = 2004, 6,30,0,0,0, data_date(:,7) = 2004, 7,07,0,0,0, data_date(:,8) = 2004, 7,14,0,0,0, layer_type = 'SFC', opt_periodic_year = .false.,/###### <--- PLEASE CHECK THE DATES BEFORE RUNNING!!!
25
nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)
• ktype = ‘3D’, ‘2D’, ‘GO’ (ocean), ‘GL’(land), ‘ISCCP’
&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360,/&NMHIST ktype = ‘2D’, item = ‘sl_slp’, file = ‘sa_slp’, output_type = ‘AVERAGE’, step = 120,/
variable namefile name ( ms_tem.rgn?????)
‘SNAPSHOT’or
‘AVERAGE’
timestep interval of outputif DTL=30s, step = 120: 1 hourly step = 360: 3 hourly
26
nhm_driver.cnf (Output Variables)nhm_driver.cnf (Output Variables)
• &NMHISD: default setting of &NMHIST
• Output with pressure levels
&NMHISD direct_access = .TRUE., no_vintrpl = .false.,/
.true. for z* coordinate
.false. for z coordinate
&NMHIST ktype = ‘3D’, item = ‘ml_tem’, file = ‘ms_tem’, output_type = ‘SNAPSHOT’, step = 360, out_prelev = .true.,/
&NMHISD... npreslev = 23, pres_levs = 1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 70, 50, 30, 20, 10, 7, 5, 3, 2, 1, 0.4,/
27
Run NICAM !Run NICAM !
• See tutorial package for more detailed descriptions of nhm_driver.cnf
% mpirun -np 320 ${TOPPATH}/bin/nhm_driver
Output: ms_tem.rgn?????, sa_slp.rgn?????, .....(huge number of files are created)
28
Icosahedral Data Icosahedral Data Lat-Lon Data Lat-Lon Data
• ico2ll.cnf&ICO2LL_PARAM glevel = 10, rlevel = 5, input_dir = './icodata/', output_dir = './lldata/' info_fname = 'history.info', llmap_dir = '../../grid/llmap/data/' llmap_base = 'llmap', lon_swap = .true., input_size = 4, init_year = 2004, init_month = 6, init_day = 1,/
% ${TOPPATH}/bin/ico2ll
automatically generated by nhm_driver
Output: ms_tem.ctl, ms_tem.grd, sa_slp.ctl, sa_slp.grd, .....
29
Restarting RunRestarting Run
• Followings should be changed in nhm_driver.cnf:– &TIMEPARAM: start_*
– External condition• &RESTARTPARAM
• &NM_OCEAN_INIT
• &NM_LANDINIT
• &DIAGVARPARAM
30
Changing GlevelChanging Glevel
• Followings should be changed in nhm_driver.cnf:– &RUNCONFPARAM (e.g. cloud resolving / cloud param.)
– &ADMPARAM• glevel, rlevel, rgnmngfname
– &NUMFILTERPARAM (as glevel increase by 1,)• Kh_coef_maxlim , Kh_coef_minlim, alpha_d : divided by 8
• gamma_h_lap1 : divided by 2
– &TIMEPARAM• dtl, lstep_max (if necessary)
– &NM_RD_TIME• TINTV= 10 mins for gl9, but 5 mins for gl10 and 11
– File names
31
Handling Runtime ErrorHandling Runtime Error
• Model bug? Configuration mistake? Numerical instability?• Model run diagnosis
– msg.????? : message from NICAM– standard out/err
• Check List– Where is the run fail? What module/subroutine/line?– Are the prescribed files such as external_data and initial_data correct?
• check them using ico2ll• unit of variables• glevel/rlevel• file name, little/big endian
– compiler option (such as “-assume byterecl” in intel ifort)– memory enough for the run?– parameters in nhm_driver.cnf correct?– Does the error also occur in the lower glevel/rlevel?
If the runtime error seems to occur due to numerical instability, ...
32
Numerical Filter in nhm_driver.cnfNumerical Filter in nhm_driver.cnf&NUMFILTERPARAM dep_hgrid = .false. DEEP_EFFECT = .false., hdiff_fact_rho = 0.01D0, hdiff_fact_q = 0.0d0,
hdiff_type = 'NONLINEAR1', cfact = 2.0D0, Kh_coef_maxlim = 3.125D+12, Kh_coef_minlim = 3.125D+11, ZD_hdiff_nl = 20000D0,
gamma_h_lap1 = 6.25D+5, ZD_hdiff_lap1 = 20000.0D0,
divdamp_type = 'DIRECT', lap_order_divdamp = 2, alpha_d = 3.125D+11, alpha_dv = 0.00D0,
gamma_v = 0.0D0,
ZD = 25000.0d0, alpha_r = 0.0D0,/
horizontal diffusion
Laplacian-type horizontal diffusion
3D divergence damping
vertical diffusion
Rayleigh friction
In addition, 2D divergence damping
33
Other Configurations in nhm_driver.cnfOther Configurations in nhm_driver.cnf
&nm_tb_my2moist ICE = .true., LENGTH_TYPE = '3KM', NU_MIN = 0.1D0, KH_MIN = 0.1D0, N2max = 0.0016, /##### <--- If unstable, reduce the N2max.
&TIMEPARAM INTEG_TYPE = 'RK2', LSTEP_MAX = 8640, SSTEP_MAX = 4, DTL = 30.0D0, SPLIT = .TRUE., start_year = 2004, start_month = 6, start_day = 1,/##### <--- default : RK2 with SSTEP_MAX = 4;##### <--- But, if unstable, use RK3 with SSTEP_MAX=6.
34
Adjusting Initial ConditionsAdjusting Initial Conditions
• Tool from Tatsuya Seiki (CCSR, Univ. Tokyo)– modify pressure to achieve hydrostatic balance
– modify vertical velocity using continuity equation
35
SummarySummary
• Please inform us if you find bugs, misleading points, suggestions, etc in NICAM tutorial package.
Thank you
38
0. Who am I ?0. Who am I ?
• Research interest– storm track
– stratosphere dynamics
– global warming
– role of cloud in climate system
• Misc– GrADS script library
40
Generating Vertical GridGenerating Vertical Grid
• mkvlayer.cnf&MKVLAYER_CNF num_of_layer = 40, layer_type = 'GIVEN', infname = './vgrid40.txt', outfname = './data/vgrid40.dat',/
0.000000000000000E+000161.683000000000335.958000000000523.806000000000726.285000000000944.5340000000001179.781000000001433.349000000001706.667000000002001.272000000002318.822000000002661.104000000003030.045000000003427.720000000003856.368000000004318.400000000004816.418000000005353.223000000005931.837000000006555.516000000007227.769000000007952.380000000008733.428000000009575.3060000000010482.753000000011460.876000000012515.179000000013651.596000000014876.521000000016196.850000000017620.010000000019154.012000000020807.488000000022589.744000000024510.811000000026581.496000000028813.457000000031219.252000000033812.418000000036607.555000000039620.3830000000
TODO: how to create refstateorNo explanation of vertical grid
42
nhm_driver.cnfnhm_driver.cnf&nm_rd_time
TINTV=10.0D0,TUNIT='MIN'
/###### <--- TINTV = 10min for gl9, but 5min for gl10 and 11.
43
nhm_driver.cnf (Resolution, Grid)nhm_driver.cnf (Resolution, Grid)
&ADMPARAM glevel = 10, rlevel = 5, vlayer = 40, rgnmngfname = '../../mnginfo/data/rl05-prc320.info',/&GRDPARAM hgrid_fname = '../../grid/hgrid/data/grid', vgrid_fname = '../../grid/vgrid/data/vgrid40.dat', topo_fname = '../../external_data/MATSIRO/data/topog/topog', toposd_fname = '../../external_data/MATSIRO/data/topog_std/topog_std', vegeindex_fname = '../../external_data/MATSIRO/data/veget_mat/veget_mat'/