seadas training course nasa ocean biology processing group *** venue line 1 *** *** venue line 2 ***...
Post on 01-Apr-2015
232 Views
Preview:
TRANSCRIPT
SeaDAS Training CourseNASA Ocean Biology Processing Group
*** Venue line 1 ***
*** Venue line 2 ***
***Date***
-The SeaWiFS Data Analysis System (SeaDAS) is
a comprehensive image analysis package for the
processing, display, analysis, and quality control
of ocean color data.
-SeaDAS is intended for use with all of the OBPG
supported sensors: MODIS/Aqua, MODIS/Terra,
SeaWiFS, OCTS, CZCS.
-SeaDAS can also be very useful as a general
scientific imagery/data analysis package.
What is SeaDAS?
SeaDAS - Processing and Display
Processing-All sensors L0 or L1A to SMI-subscening-browse products
Data Visualization-image package (reads many formats)-geolocated display (coastline, gridlines, etc.)-various image/data functions
Data Manipulation-projection-band operations-output data
SeaDAS Functionalities
-SeaDAS runs on “UNIX-like” systems (UNIX/linux/OS X)
-This means it does NOT run on Windows.
-Dedicated UNIX computer best, but there are options for
Windows users.
UNIX
linux
Mac OS X
SGI IRIX
Sun Solaris
MicrosoftWindowsUNIX
-UNIX is a fully featured modern Operating System
-History (AT&T Bell Labs, Berkeley, UNIX, linux, OS X)
-It is available in a variety of ‘flavors’
-It is comprised of simple tools (commands) that perform a
single function well
-These tools can be used together to perform complex tasks
-Powerful Command line
-GUI
What is UNIX?
Sample linux directory structure
The UNIX Command Line
UNIX Directory Structure:
-At login, you are in your home directory.
-This is your “current working directory”.
-All files and directories below your home
directory usually belong to you.
-You change your working directory by changing
to another directory with the “cd” command.
UNIX Basics
Binaries and Libraries
-binaries are executable files created by compiling code
-mainly used for data processing (e.g. msl12 = L1->L2 processing)
-made from C and/or Fortran code
-SeaDAS also includes HDF binaries and libraries
UNIX Shell Scripts
-Shell scripts are executable text files containing many commands
-mainly used as “wrappers” for processing
-there are also stand-alone utility scripts
IDL Code
SeaDAS is made up of binaries and libraries, UNIX shell scripts, and IDL code.
-IDL stands for the Interactive Data Language
-Made by ITT Visual Information Solutions, ITT VIS (formerly RSI)
-It is a scientific programming language (similar to Matlab)
-It is also a “scientific environment”
-SeaDAS GUI written in IDL
-SeaDAS also uses IDL for data display and analysis functions
-non-SeaDAS IDL functionalities are also available!
Runtime SeaDAS - free, but no SeaDAS command line or procedures
Purchasing IDL - command line, procedures, full IDL environment
IDL (Interactive Data Language)
SeaDAS Directory Structureseadas/ - - - - - - - - - - - - SeaDAS root directory bin/ - - - - - - - - - - C and Fortran processing binaries config/ - - - - - - - - - SeaDAS config files (Imake and default files) data/ - - - - - - - - - - Data files used for processing and display common/benchmarks/ - Data files/scripts for running SeaDAS benchmarks common/luts/ - - - - Color LUTs directory (store custom LUTs here) common/metoz/ - - - MET/OZONE ancillary files storage directory common/oisst/ - - - OISST ancillary files storage directory modis/atteph/ - - - MODIS attitude and ephemeris storage directory demo/ - - - - - - - - - - Demo and test data files doc/ - - - - - - - - - - SeaDAS HTML documentation files etc/ - - - - - - - - - - Stand-alone data processing wrapper scripts examples/ - - - - - - - - Example batch scripts hdf/ - - - - - - - - - - HDF software (binaries and libraries) idl_lib/ - - - - - - - - Copy of all SeaDAS IDL routines idl_rt/ - - - - - - - - - Required directory for Runtime SeaDAS mode inc/ - - - - - - - - - - Include files for C and Fortran routines lib/ - - - - - - - - - - C and Fortran compiled libraries src/ - - - - - - - - - - Root directory of all SeaDAS source code c4idl/ - - - - - - - C routines used within IDL programs c_procs/ - - - - - - Main data processing C routines idl_procs/ - - - - - IDL routines idl_utils/ - - - - - IDL utility routines lib/ - - - - - - - - C and Fortran routines for SeaDAS libraries utils/ - - - - - - - Utility programs used by wrapper scripts util/ - - - - - - - - - - SeaDAS general utilities directory var/ - - - - - - - - - - Miscellaneous ancillary navigation files
Operating Systems:
-linux (Red Hat, Fedora, CentOS, etc.)
-Mac OS X 10.3, 10.4 (PPC or Intel)
-Sun Solaris 2.7, 2.8
-SGI IRIX 6.5
-Vmware on Windows / Knoppix Live CD
RAM: 128MB minimum, 1GB+ best
Display: 1024x768 minimum, millions of colors
Disk space: 1.5GB for SeaDAS install plus space for data
Optional: full IDL license
Optional: gcc and fortran compilers
SeaDAS Requirements
SeaDAS Benchmarks (times are cpu times in seconds)
PLATFORM L1A->GEO L1A->L1B L1B->L2 L2->L3 SMI TOTAL T IME
Apple MacBook Pro 2.2 Laptop CPU: 2.16GHz Intel Core 2 Duo RAM: 2GB OS: OS X 10.4.8 (Intel)
8 18 87 5 118
Sun Fire V40z CPU: Dual 2GHz dual core AMD Opteron 870 RAM: 8GB OS: Rocks 4.1
24 30 146 12 212
Generic PC CPU: Dual Pentium 4 3.0Ghz RAM: 2GB OS: Fedora core 4
32 35 150 24 241
Apple Power Macintosh G5 CPU: Dual 2.5GHz PowerPC G5 RAM: 4GB OS: OS X 10.3.9 (PPC)
18 31 183 22 254
Generic PC CPU: Dual 1.3GHz Pentium III RAM: 1GB OS: Redhat Linux 7.3
29 72 325 26 452
There are 3 ways to use SeaDAS:
1. Graphical User Interface (GUI)
2. Command Line:
-IDL command line
-UNIX command line
3. Automated:
-IDL command (batch) files
-IDL procedures
-UNIX shell scripts
SeaDAS Modes
On the server:
-Change directory into your home (~/) directory: cd
-List the contents of your home (~/) directory: ls -l
Change directory into your home (~/) directory: cd $SEADAS
-List the contents of the SeaDAS directory: ls -l
-List the contents of the SeaDAS bin directory: ls bin
-List the contents of the SeaDAS etc directory: ls etc
-List your shell environment variables: env
-change directory into your data directory: cd data
-List the contents of your data directory: ls -l data
Exercise
Understanding the SeaDAS Environment
-On the server:
cd ~/data/process seadas
Display the SeaWiFS Level 1 subscene S2004111164518.L1A_MLAC.x.hdf:
-Click Display -Select the S2004111164518.L1A_MLAC.x.hdf file and click Ok -Select the 555nm product and click Load -In the Band List Selection window click Display -Resize display window by dragging the lower right corner -Using the Functions menu: *overlay a Coastline and latitude/longitude Grid *turn on the Color Bar *turn on the Cursor Position window *Select Color LUT->Load LUT and select different color LUTs
ExerciseDisplaying data with SeaDAS
Review of OBPGSatellite Data
Data Processing
Data Processing usingthe SeaDAS GUI
Band Number
Wavelength (nm)
Band Width
(nm)
Spatial Resolution
(m)
SNR at Ltyp
Ltyp mW cm-2
m-1 sr-1
Lmax mW cm-2
m-1 sr-1 8 412 15 1000 1773 7.84 26.9 9 443 10 1000 2253 6.99 19.0
10 488 10 1000 2270 5.38 14.0 11 531 10 1000 2183 3.87 11.1 12 551 10 1000 2200 3.50 8.8 13 667 10 1000 1962 1.47 4.2 14 678 10 1000 2175 1.38 4.2 15 748 10 1000 1371 0.889 3.5 16 869 15 1000 1112 0.460 2.5
Operational MODIS Ocean Band Suite
Band Number
Wavelength (nm)
Band Width
(nm)
Spatial Resolution
(m)
NEdT
22 3959 60 1000 0.07 23 4050 60 1000 0.07 31 11000 60 1000 0.05 32 12000 60 1000 0.05
VIS/NIR
Ocean Color
Thermal
SST
Data Levels & Flow• Level 0
– raw digital counts– native binary format
• Level 1A – raw digital counts– HDF formatted
• Level 1B
– calibrated reflectances
– converted telemetry
• Level 2
– geolocated geophysical products for each pixel
• Ancillary data– wind speed– surface pressure – total ozone– Reynolds SST
• GEO – geolocation– radiant path geometry
• ATT & EPH – spacecraft attitude– spacecraft position
Data Levels & Flow• Level 0
– raw digital counts– native binary format
• Level 1A – raw digital counts– HDF formatted
• Level 1B
– calibrated reflectances
– converted telemetry
• Level 2
– geolocated geophysical products for each pixel
• Ancillary data– wind speed– surface pressure – total ozone– Reynolds SST
• GEO – geolocation– radiant path geometry
• ATT & EPH – spacecraft attitude– spacecraft position
• Level 1A Subset – reduced to standard
ocean bands only
L0 to L1A
1. Display A2005205180000.L1A (choose EV_3959_22)
2. Functions -> Cursor Position
3. Process -> MODIS -> modis_subscene
4. Enter SW-Corner: 40, -75 / NE-Corner: 43, -65
L1A Subscene
L1A to L1B
MSL12Multi-Sensor Level-1 to Level-2
• Common software for Level-2 processing of MODIS, SeaWiFS and other sensors in a consistent manner.
• Supports a multitude of product algorithms and processing methodologies.
– standard and non-standard, validated and experimental
– run-time selection of output product suite
• For MODIS, replaced:
– PGE09, PGE10
– products MOD18 - MOD28
L1B to L2(msl12)
-As data is processed by msl12 from Level 1 to Level 2, checks are made for different defined conditions.
-When certain tests and conditions are met for a given pixel, a flag is set for that pixel for that condition.
-A total of 31 flags can be set for each pixel.
-These msl12 Level 2 processing flags are stored in the Level 2 data file as the "l2_flags" product.
-The storage method sets bits to 0 or 1 in 32-bit integers that correspond to each pixel.
Level 2 Processing Flags
Level 2 Processing Flags
(flags in red are masked during Level 3 processing)
ChlorophyllRGB Image
Glint
Sediments
Cloud
Level-2 Flags and Masking
nLw (443)RGB Image
Glint
Sediments
Cloud
Level-2 Flags and Masking
Add masking for high glintAdd masking for straylight
-For Level 1 to Level 2 processing (msl12), masked pixels are not processed and are typically set to zero so as to eliminate them from future analysis.
-For products where zero could be a valid data value, a number outside the possible data range is substituted.
-For SeaWiFS and MODIS, msl12 currently has 8 predefined L1A processing masks (each comprised of only one flag) that can be turned on (1) or off (0) by the user. These 8 masks and their default settings are:
maskland=1maskcloud=1maskhilt=1maskbath=0maskglint=0masksatzen=0masksunzen=0maskstlight=0
Processing Masks
SeaDAS Level 2 Flags Display Function
SST Quality Tests
SST Quality Levels
SeaDAS Data Functions
Importing and ExportingData using SeaDAS
SeaDAS can export data
files as HDF, SeaDAS
Mapped, flat binary, ascii
text, png, and postscript
-All OBPG HDF Data files (MODIS, SeaWiFS, OCTS, CZCS)
-Some non-OBPG legacy Ocean HDF formats
-General HDF (any 2D SDS in an HDF file)
-Navigated HDF (a 2D SDS plus a “lat” SDS and “lon” SDS)
-“SeaDAS Mapped” HDF files
-AVHRR V5 w/ standard NASA or NOAA SDS attribute names
-AVHRR V4 SST/JPL PO.DAAC HDF Pathfinder Best SST
-AVHRR V4 SST/JPL PO.DAAC HDF MCSST
-Binary flat files
-PNG files
-True Color files
-Multi-column ascii text files
SeaDAS can read / display / analyse:
Data Product Geographic Resolution,
Mapping and Binning
Definitions
Projection - any process which transforms a spatially organized data set from one coordinate system to another. For MODIS Ocean data products this will refer to two-dimensional data sets.
Mapping - a process of transforming a data set from an arbitrary spatial organization to a uniform (rectangular, row-by-column) organization, by processes of projection and resampling.
Binning - a process of projecting and aggregating data from an arbitrary spatial and temporal organization, to a uniform spatial scale over a defined time range. Ideally the binning process will preserve both the central tendency (e.g., average) and the variation in the data points that contribute to a bin.
Ocean Product Projections
Equal-area: sinusoidal, with equally space rows and number of bins per row proportional to sine of latitude; resolutions of 4.6 and 9.2 km
Equal-angle: rectangular (Platte Carre) with rows and columns equally spaced in latitude and longitude; resolutions of 24 and 12 points per degree.
Ocean equal-area and equal-angle projections are equivalent at the equator.
Sinusoidal Equal-Area Projection
Level-3 Binned vs Mapped
bin file grid
map file grid
• bin files
– multiple products– stored as float– sampling statistics included
• map files
– single product– stored as scaled integer
Increasing Pixel Size
MODIS “Bowtie” Effect
One MODIS scan at ~45 degrees scan angle
Two MODIS scans showing overlap of pixels
Multiple MODIS scans showing pixel overlap
Bin boundaries overlaid on pixel locations
Ocean coverage over time for binned files
Binned and Mapped Products
Binned products: generated daily from Level-2, and then aggregated to:
-8 days -monthly -seasonal -annual -mission
Standard mapped image (SMI) products are generated at each temporal resolution by projecting binned files to the equal-angle grid.
Level-3 Processing
• Level 3 binned
– geophysical products averaged spatially and/or temporally
– sinusoidally distributed, equal area bins
• Level 3 mapped
– images created by mapping and scaling binned products
– user-friendly, cylindrical equiangular projection
• Level 2
– geolocated geophysical products for each pixel
Standard Products
• Bin resolution 4.6 x 4.6 km2
• Mapped resolution
– 0.042-deg– 0.084-deg
• Composite Periods
– Daily– 8-day– Monthly– Seasonal– Yearly– Mission
Level 2 Processing Flags
(flags in red are masked during Level 3 processing)
SeaDAS Command Line Processing
-All SeaDAS processing can also be done using the SeaDAS
(IDL) command line.
-Virtually everything you can do via the GUI can be done
using the SeaDAS command line.
-IDL commands can be intermixed with SeaDAS commands.
-If you haven’t purchased IDL, you can still use command
(batch) files to access the command line functionality.
SeaDAS/IDL Command Line
SeaDAS> modis_l1bgen,/help
Usage: modis_l1bgen, ifile="ifile", geofile="geofile", ofile="ofile",
filehkm="filehkm", fileqkm="fileqkm",
refllutfile="refllutfile", emislutfile="emislutfile",
qalutfile="qalutfile",
delete1km=delete1km, deletehkm=deletehkm, deleteqkm=deleteqkm,
savelog=savelog, wait=wait
(Note use of quotes around path/filenames)
Example:
modis_l1bgen, ifile="$SDSDEMO/A2005191123500.L1A_LAC",
geofile="./A2005191123500.GEO", /wait
SeaDAS>
SeaDAS/IDL Command Line
SeaDAS> print, “Hello World!”
Hello World!
SeaDAS>
SeaDAS> modis_l1bgen, ifile="A2005205180000.L1A_sub”,
geofile="A2005205180000.GEO_sub", /wait
% seadas -em -b simple_commands.txt
ExampleProcessing using the SeaDAS command line
% modis_L1A_to_L1B.csh
Usage: modis_L1A_to_L1B.csh L1A_file GEO_file [OPTIONS]
Options:
-o L1B_1KM_filename Output MODIS L1B 1KM HDF filename
-h L1B_HKM_filename Output MODIS L1B HKM HDF filename
-q L1B_QKM_filename Output MODIS L1B QKM HDF filename
-rlut REFL_LUT_filepath Full path and filename of Reflective LUT file
-elut EMIS_LUT_filepath Full path and filename of Emissive LUT file
-qlut QA_LUT_filepath Full path and filename of QA LUT file
-delete-1km Delete 1km resolution L1B file
-delete-hkm Delete 500m resolution L1B file
-delete-qkm Delete 250m resolution L1B file
-save-log Save the Level 1B processing log
%
UNIX Command Line
Level 1B processing example:
% modis_L1A_to_L1B.csh A2005205180000.L1A_sub \
A2005205180000.GEO_sub -delete-hkm -delete-qkm
ExampleProcessing using the UNIX command line
Level 2 processing example:
% ms_met.csh A2005205180000.L1B_LAC
% ms_ozone.csh A2005205180000.L1B_LAC
% ms_oisst.csh A2005205180000.L1B_LAC
...this will create three files:
A2005205180000.L1B_LAC.met_list
A2005205180000.L1B_LAC.ozone_list
A2005205180000.L1B_LAC.sst_list
...then you could run msl12 on the UNIX command line:
% msl12 \
par=A2005205180000.L1B_LAC.met_list \
par=A2005205180000.L1B_LAC.ozone_list \
par=A2005205180000.L1B_LAC.sst_list \
ifile=A2005205180000.L1B_LAC \
geofile=A2005205180000.GEO_sub \
ofile1=A2005205180000.L2
Example (cont’):Processing using the UNIX command line
SeaDAS can be completely automated using:
-IDL command files, and/or
-IDL procedures, and/or
-UNIX shell scripts
For certain things it makes sense to use IDL, other things
can be done more simply and quickly using UNIX.
Automating SeaDAS Processing
;;; this is an IDL command file ;;;
; modify "*L2_LAC" to match the filename pattern of your L2 files
filenames=findfile('*L2_LAC')
for i=0, n_elements(filenames)-1 do begin & $
base=strmid(filenames,0,14) & $
outfile = base+'.L3b' & $
print, "Running l2bin on ", filenames(i) & $
l2bin, ifile=filenames(i), $
ofile=outfile(i), resolve=4, $
FLAGUSE='ATMFAIL,LAND,HILT,HISATZEN,STRAYLIGHT,CLDICE,COCCOLITH,
HISOLZEN,LOWLW,CHLFAIL,NAVWARN,ABSAER,MAXAERITER,CHLWARN,ATMWARN,
NAVFAIL,FILTER' & $
endfor
print, "Finished!”
ExampleUsing an IDL command file to automate processing
#!/bin/shfor FILE in *L2_LACdo BASE=`echo $FILE |awk -F. '{ print $1 }'` echo "Running l2bin on $FILE" l2bin infile=$FILE ofile=$BASE.L3b resolve=4 >/dev/nulldone
ls -1r *.L3b >list.txtecho "Running l3bin"l3bin in=list.txt out=M20052022005205.L3b noext=1 >/dev/null
echo "Running smigen"smigen ifile=M20052022005205.L3b prod=chlor_a \ofile=M20052022005205_smi_CHLOR_A_auto.hdf \resolution=4km latnorth=48.0000 latsouth=35.0000 \lonwest=-78.0000 loneast=-62.0000
echo "Finished!"
ExampleUsing a UNIX shell script to automate processing
fin
When you are assigned a guest # and server, please log onto your server and test SeaDAS:
ssh -X guest#@192.168.1.### username: guest# password: guest123
Execute: seadas
SeaDAS Server Login
MERIS Support for Processing and Display
top related