http:// slides based on lectures from the ncl workshop also see the ncl homepage
TRANSCRIPT
http://www.ncl.ucar.edu/Training/Workshops/lectures.shtm
http://www.ncl.ucar.edu/
Slides based on lectures from the NCL workshop
Also see the NCL homepage
http://www.atmos.albany.edu/student/janiga/temp/incoming_grad_lecture.ppt
This ppt can be downloaded off the web at the below address
For additional help…
http://www.ncl.ucar.edu/Support/ncl_talk.shtml
The NCL variable model is based on netCDF
netCDF filesself describing
(ideally) all info contained within the file no external information needed to determine file
contentsportable [machine independent]
Similar to HDF (NASA) and GRIB (NCEP, ECMWF, etc) formats; all these are read by NCL using the same one line.
Supported by many software tools / languages NCL, IDL, Matlab, ferret, GrADS, F, C, C++,Java command line file operators: NCO, CDO
perform various tasks (i.e. concatenation) very efficiently
http://nco.sourceforge.net http://www.mpimet.mpg.de/fileadmin/software/cdo
ncview quick visualization using gui if in COARDS format:
IDV, NCVIEW, Panopoly, etc.
Examining a netCDF file
• ncdump file_name dumps the entire contents of a file; prints every value
• ncdump -h file_name Dumps header information [most commonly used] NCL equivalent: ncl_filedump file_name
• ncdump -v U file_name NCL equivalent: ncl_filedump –v U file_name
• ncl_filedump file_name [ more general ] netCDF3/4, GRIB-1,GRIB-2, HDF, HDF-EOS [HDF5]
• ncview file_name [visualize file contents]
Now for an example….
Parts of netCDF file
DIMENSION SIZES dimensions: lat = 64 lon = 128 time = 12
VARIABLES: Names , Types, Attributes, Coordinate Variablesvariables: float lat(lat) lat:long_name = "latitude" lat:units = "degrees_north" float lon(lon) lon:long_name = "longitude" lon:units = "degrees_east" int time(time) time:long_name = "time" time:units = "Month of Year" double T(time, lat, lon) T:long_name = “Temperature” T:units = “C" T:missing_value = 1.e+20f T:_FillValue = 1.e+20f
FILE ATTRIBUTESglobal attributes: title = “Temp: 1999” source = “NCAR” conventions = "None”
Longitude coordinate variable (1D, &)
Latitude coordinate variable (1D, &
)
attributes @:
• long_name
• units
visual: simple 2D netCDF Variable
coordinate variables (rectilinear grid)
Non-Traditional Grids or Arrays• netCDF and NCL can handle various “other” types of data:
Spectral coefficents (YOTC) Lambert conformal (WRF)
http://www.ncl.ucar.edu/Applications/wrflc.shtml Radiosondes Surface station data Hurricane tracks
http://www.ncl.ucar.edu/Applications/unique.shtml Make your own!
• Tons of pre-defined functions for manipulating and converting between different formats as well as plotting.http://www.ncl.ucar.edu/Document/Functions/list_alpha.shtm
netCDF [NCL] Variable model
f = addfile(“foo.nc”,”r”) ; grb/hdfx = f->X
XScalar
or Array
attributeslong_name_FillValue
unitsadd_offset
scale_factoretc. values
Scalar or
Array
attributeslong_name_FillValue
unitsadd_offset
scale_factoretc.
accessed via @ accessed via &
timelevlatlonetc.
coordinates timelevlatlonetc.
coord var
NCL reads the scalar/array, attributes, and coordinate
variables as an object
X
Detailed Look netCDF Variable (NCL)ncl <return> ; interactive mode
ncl 0 > f = addfile ("UV300.nc", "r") ; open file
ncl 1 > u = f->U ; import STRUCTURE
ncl 2 > printVarSummary (u) ; overview of variable
Variable: uType: floatTotal Size: 65536 bytes 16384 valuesNumber of Dimensions: 3Dimensions and Sizes: [time | 2] x [lat | 64] x [lon | 128]Coordinates: time: [ 1 .. 7 ] lat: [ -87.8638 .. 87.8638 ] lon: [ 0 .. 357.185]Number of Attributes: 5 _FillValue : 1e36 units : m/s long_name : Zonal Wind Component short_name : U missing_value : 1e36
Classic netCDFVariable Model
NCL syntax/funcs query use modify add any aspect of data object
NCAR Command Language• Complete Programming Language
data types variables operators expressions conditional statements loops functions/procedures/graphics
• Features query / manipulate meta data import data in a variety of formats array syntax / operations can use user fortran/C codes and commercial libraries most functions/procedures ignore missing data
Running NCL• Interactive Mode (Command line)
ncl [options][command-line-arguments] <return> ncl> enter commands ncl> quit <return>
can save interactive commandsncl> record “file_name” ncl> stop record
• Batch Mode [ .ncl suffix is optional] ncl [options][arguments] script.ncl
ncl < script.ncl [also acceptable] nohup ncl [options][arguments] script.ncl > out.txt &"&" means put in background“nohup” means don’t quit when I logoff (ie long scripts).
NCL predefined options• ncl –hnxV [predfined options are preceded by dash]• may be used for interactive or batch mode • informational
ncl –h [display predefined options and usage and exit] ncl –V [print the NCL version and exit]
• action ncl –x [echo statements as encountered (debug)] ncl –n [don't enumerate dimensions of values in print() ]
• multiple predefined options ncl –nx [ not ncl –n –x ]
http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclCLO.shtml
Variables can be set on the command line prior to a script. However, this tends to not be necessary.Input from a text file or systemfunc()
Sample Batch Script: sample.ncl load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl“load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl“
f1 = addfile("TMP_58-97.nc", "r") ; open netCDF file T = f1->Tmp ; T(time,lev,lat,lon) =>(480,17,73,144) f2 = addfile(“P_1958-1997.grb", "r") ; open GRIB file P = f2->Pres ; P(time,lev,lat,lon) f3 = addfile(“Q_1958-1997.hdfeos", "r") ; open hdf –eos file
Q = f3->Specific_Humidity ; Q(time,lev,lat,lon)
wks = gsn_open_wks("ps", “sample") ; open a graphic workstation gsn_define_colormap (wks,"gui_default") ; change from default
plot = gsn_csm_contour_map_polar (wks,T(0,5,:,:),False) res = True ; change default plot res@cnFillOn = True ; use colors res@gsnSpreadColors = True ; use entire color map plot = gsn_csm_pres_hgt (wks,T(0,:,{50},:), res )
pot = T* (1.+0.622*Q) *(1000/P)^0.286 ; potential temperature (array)
print("pot: min="+ min(pot) + " max="+ max(pot) )
begin ; optional
end ; only if begin is present
Outline: NCL Language Basics
• special syntax characters• data types• Variables netCDF/NCL variable model • attributes • _FillValue• named dimensions• coordinate variables• print and printVarSummary• shaping • subscripting
NCL Syntax Characters• ; - comment [can appear anywhere]• @ - reference/create attributes• ! - reference/create named dimension• & - reference/create coordinate variable•{…} - coordinate subscripting• $ - enclose strings when (im/ex)port variables via addfile•(/../) - array construct characters• : - array syntax• | - separator for named dimensions• \ - continue character [statement to span multiple lines]• :: - syntax for external shared objects (eg, fortran/C)• -> - use to (im/ex)port variables via addfile function•“ “ - use for a string
Data TypesNumeric• double (64 bit)• float (32 bit)• long (32 or 64 bit)• integer (32 bit)• short (16 bit)• byte ( 8 bit, 0-255)• complex NOT supported
non-Numeric• string• character• graphic• file• logical• list
Time coordinate variables frequently double.
If IR data (K) was offset by 100 it could be stored in byte instead of double (8 times less disk space) or float (4 times less disk space).
Data will often be packed into short or byte this way xFloat = short2flt(xShort)xShort@scale_factorxShort@add_offset
Attributes
• assign/access with @ character T@units = “deg C” T@long_name = “temperature” T@whatever = an array of any dimension or type T@_FillValue = -999. plot title uses x@long_name by default
• attribute functions [isatt, getfilevaratts] if ([email protected].”K”) then
• delete can eliminate an attribute delete(T@title)
• info about a variable or file [meta data] attributes can be any data type except file scalar, multi dimensional array (string, numeric)
_FillValue• Unidata and NCL reserved attribute • most NCL functions ignore _FillValue
• netCDF Operators [NCO]: _FillValue attribute• ncview: missing_value attribute
best to create netCDF files with both
• NCL: don’t use commonly encountered values as a _FillValue!
• Note: “missing_value” attribute: no special status to NCL if “T” has “missing_value” attribute but no “_FillValue ”
NCL creates: T@_FillValue = T@missing_value use built-in function “ismissing” to check for _FillValue
if (any(ismissing(T))) then T = where (ismissing(T),0,T)
Arrays
• row major ….. like C/C++ left dimension varies slowest; right varies fastest dimension numbering left to right [0,1,..]
• indicies [subscripts] are zero based [0,N-1]
Consider T(:,:,:) left dimension is 0 [ T!0 ] middle dimension is 1 [ T!1 ] right dimension is 2 [ T!2 ]
NCL (netCDF) Dimensions
• assigned with ! character {let T(:,:,:)} T!0 = "time" ; leftmost [slowest varying] dim T!1 = "lat“ T!2 = "lon" ; rightmost [fastest varying] dim
• dim names may be renamed, retrieved T!1 = "LAT" … dName = T!2
• delete can eliminate: delete (T!2)
• Named dimensions used to reshape
• may be “named”provides alternative way to reference subscriptsrecommendation: always name dimensionsuse NCL syntax
NCL (netCDF) Coordinate Variables• data coordinates [ eg: time, level, lat, lon ]
strict netCDF definition 1D numeric array monotonically in[de]creasing numeric values can only be assigned to a named dimension 1D array must be the same size as dimension should not have _FillValue attribute
coordinate values assigned via & character T&lat = latitude ; Here latitude is a 1D array
• used in graphics, coordinate subscripting
• coordinate functions iscoord , isfilevarcoord
• delete can eliminate coordinate arraydelete(T&time)
Create and Assign Coordinate Variables
• assign values to named dimension time&time = time lon&lon = lon
• let x be 2D: name dimensions x!0 = “time” … x!1 = “lon”
• assign coordinate variables to x x&time = time … x&lon = lon
• assign dimension name [same as variable name] time!0 = “time” lon!0 = “lon”
• create 1D array time = (/ 1980, 1981, 1982 /) ; integer lon = (/ -90, ... 90 /) ; float
Access/Change/Create/Delete Meta Data• @ attributes
u@long_name = "U" lonName = u@long_name
• ! named dimensions u!0 = "TIME" tName = u!0
• & coordinate variable u&lat = (/ -90., -85, .... , 85., 90. /) latitude = u&lat
• $ substitute string x = fin->$variable(n)$(date,{-30:60},:)
Variable Assignment
• Value-only assignment (no meta copy) U multi-dimensional array with meta data
Uval = (/ U /) or Uval = (/ f->U/) the (/ ... /) operator pair strips meta data
• Variable-to-Variable assignment consider y = x where x is previously defined
if y not defined: y has same type/shape/values/meta data as x
if y predefined: y must have same shape and type or, x must be coerceible to the type of y y attributes, dimension names and coordinate variables, if they exist, will be over written
Variable Subscripting (1 of 3)
Standard Array Subscripting• ranges: start/end and [optional] stride• indices separated by :• omitting start/end index implies default begin/end
Consider T(time,lat,lon)T entire array [ don't use T(:,:,:) ]T(0,:,::5) 1st time index, all lat, every 5th lon T(0,::-1,:50) 1st time index, reverse lat order, 1st
51 lon T(:1,45,10:20) 1st 2 time indices, 46th index of lat,
10-20 indicies of lon
Variable Subscripting (2 of 3)
Coordinate Variable Subscripting• only applies to coordinate variables• same rules apply for ranges, strides, defaults• use curly brackets {…}• standard and coordinate subs can be mixed [if no reorder]
T(:,{-30:30},:) all times/lon, lat -30° to +30° (inclusive)
T(0,{-20},{-180:35:3}) 1st time, lat nearest - 20°, every 3rd lon between -180° and 35°
Variable Subscripting (3 of 3)
Named Dimensions• only used for dimension reordering• indicated by |• dim names must be used for each subscript• named/coordinate subscripting can be mixed
Consider T(time,lat,lon)
t = T(lat|:, lon|:, time|:) makes t(lat,lon,time)
t = T(time|:,{lon|90:120},{lat|-20:20}) all times,
90-120° lon, -20-20° lat
Longitude coordinate variable (1D)
Latitude coordinate variable (1D)
Standard and Coordinate Subscripting
Standard:
T(9:13,1:8)
Coordinate:
T({-10:20},{-170:-110})
Combined:
T({-10:20}, 1:8)
“printing”• printVarSummary
gives gross overview of a variable
• print same info as printVarSummary prints values
• write_matrix print to standard out or a file format control of numerical output can write to file also
print (1 of 2)
Variable: TType: floatTotal Size: 32768 bytes 8192 valuesNumber of Dimensions: 2 Dimensions / Sizes: [lat | 64] x [lon | 128]Coordinates:
lat: [-87.86379 .. 87.86379] lon: [ 0. 0 .. 357.1875]Number of Attributes: 2 long_name: Temperature units: C(0,0) -31.7(0,1) -31.4(0,2) -32.3(0,3) -33.4(0,4) -31.3 etc. [entire T array will be printed]
• Prints out all variable information including meta data, values T(lat,lon): print (T)
print (2 of 2)
Variable: T (subsection)Type: floatTotal Size: 256 bytes 64 valuesNumber of Dimensions: 1 Dimensions / Sizes: [lat | 64] Coordinates:
lat: [-87.86379 .. 87.86379]Number of Attributes: 3 long_name: Temperature units: C lon: 109.6875 [ added ](0) -40.7(1) -33.0(2) -25.1 (3) -20.0(4) -15.3 etc.
• can be used to print a subset of array meta data, values T(lat,lon): print( T(:,103) ) or print( T(:,{110}) )
Things that can be done (look up or ask)
• Write netCDF files (compression, making large files)• Read and write binary and ascii• Functions and procedures in NCL language (over 1000),
fortran or C (most are earth science orientated; essentially Matlab or IDV for earth science).
• Loops, conditional statements.• OPeNDAP support (query and read files directly from the
web without downloading the whole file)
• http://www.ncl.ucar.edu/Document/Functions/index.shtml