1

Status of HDF-EOS, Status of HDF-EOS, Related Software and ToolsRelated Software and Tools

Abe Taaheri, Raytheon IISHDF & HDF-EOS Workshop XIV

Champaign, ILSep. 29, 2010

2

TOOLKIT / HDF-EOS Support

• Raytheon was awarded EOSDIS Evolution & Development (EED) contract in May 2010

• Maintenance support for TOOLKIT/HDF-EOS is funded on EED through March 2015

• Current support HDF-EOS 2 & 5 SDP & MTD Toolkits HDF-EOS2 to GeoTIFF Converter (HEG) HDF-EOS plug-in for HDFView Other Toolkit/HDF-EOS related software Toolkit/HDF-EOS/HEG helpdesk

3

ECS Support of HDF-EOS

• Both HDF4 and HDF5 based flavors of HDF-EOS are part of ECS baseline.

• OS’s: Solaris (10) Linux (32 & 64-bit, including IA64) Mac OS X (Power PC and Mac

Intel)Cygwin Windows XP (MTD Toolkit, HDF-

EOS2 & 5, HEG, HDFView)

• Compilers: Fortran 77/90 , g77/pgf90 C, C++, gcc, g++

Intel Fortran (SGI Altix), gfortran (Mac Intel)

4

Downloadable Software

• DAACS are supported with – TOOLKIT/HDF-EOS & HEG deliveries during

yearly DAAC releases or patch deliveries– On demand

• Other users can download our software from – our tools’ web sites– directly from our ftp site:

ftp://edhs1.gsfc.nasa.gov/edhs

5

Downloadable Software

• SDP Toolkit (Science Data Processing Tools) • MTD Toolkit (Metadata, Time/ Date Tools)

• HDF-EOS2 (HDF4 based)• HDF-EOS5 (HDF5 based)

• heconvert (converts HDF4 - based Grid/Point/Swath structures to HDF-EOS 5 equivalents)

• DumpMetadata (Dump HDF-EOS metadata)

6

Downloadable Applications

• HEG (HDF-EOS2 to GeoTIFF conversion)• HDFView Plug-in, Java-based browser

(HDF4, HDF5, HDF-EOS 2 and 5 access)

• HE5View* (HDF5 - based browser)• EOSView* (HDF4 - based browser)

* Dropped support for OS updates on July 2006

7

Software/Tools

Brief description of TOOLKIT, HDF-EOS,

And HDFView Plug-in

8

Toolkit

• Science Data Production (SDP) Toolkit :collection of tools used by

- Science software developers who producecode to process instrument data.

- HDF-EOS users who need to• Access metadata in HDF-EOS files• Perform Time/Date conversion• Access Digital Elevation Model Data• Access Ancillary Data, such as Digital Chart

of the World database (DCW), Olson World Ecosystem files, DEM data files, etc.

9

Toolkit (cont.)

• Tools provided in SDP Toolkit:- AA (Ancillary Data Access)- CBP (CELESTIAL Body Position)- CSC (Coordinate System Conversion)- CUC (Constant and Unit Conversions)- DEM (Digital Elevation Model Access)- EPH (Ephemeris Data Access)- GCT (Geo Coordinate Transformation)- IO (Input Output (File I/O))- MEM (Memory Management)- MET (Metadata Access)- PC (Process Control)- SMF (Status Message File (Error/Status))- TD (Time and Date Conversion)

10

Toolkit (cont.)

• MTD Toolkit:

– a short version of SDP Toolkit– Tools provided:

• MET (Metadata Access)• TD (Time and Date Conversion)

– Also supported in windows

11

HDF-EOS

• HDF-EOS:– an extension to HDF which standardizes storage

of and access to common Earth Sciences data structures

• By providing a single interface to data structures common to earth science

• By providing a container for EOS inventory, archive and product specific metadata

• By establishing a consistent relationship between geolocation and science data.

12

HDF-EOS 2

• HDF4 - based, storage format for EOS standard products.

• Used operationally by: – MODIS, MISR, ASTER, Landsat, AIRS – other EOS instruments

• Support for Grid/Point/Swath structures

13

HDF-EOS 5

• HDF5 - based, a complete rewrite of HDF-EOS2 with a different user interface.

• Used operationally by EOS Aura instrument teams

• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible.

• Supports same data structures (and more: Zonal average)

• Added prefix ‘HE5_’ to HDF-EOS 2 functions to distinguish two libraries

14

HDFView Plug-in

• HDFView: A Java-based browser from The HDF Group for HDF4 and HDF5 based data.

• HDF-EOS plug-in – Extends HDFView to browse HDF-EOS data in both

versions.– Is a plug-in module and does not change The HDFView

functionality – Have identical look and feel when displaying HDF and

HDF-EOS objects – Users can access a complete breakdown of HDF-EOS

objects. e.g. grid info, projection info, dims, datafields, and attributes

15

• HEG

• GeoTIFF conversion• Multi-band GeoTIFFs• Swath to Grid conversion• Spatial subsetting, Field Subsetting• Reprojection• Stitching, sub-sampling and more ….

HEG

16

Software Releases

17

Software Releases

SDP Toolkit/HDF-EOSSDP Toolkit/HDF-EOS::

• Last release – August 2010– TOOLKIT 5.2.17 – HDF-EOS 2.17, HDF- EOS5.1.13

• with HDF4.2.5, HDF5-1.8.5, Szip2.1, zlib1.2.5

• Next release – (late 2010 – early 2011)– TOOLKIT 5.2.18 (with new 500m DEM data)– HDF-EOS 2.18, HDF-EOS5.1.14

• with new releases of HDF4 (2.6), 5 (1.8.6)

18

Software Releases

MTD Toolkit:MTD Toolkit:

• Last release – Dec 2006– Version 5.2.13

• in-sync with SDP TOOLKIT 5.2.13• with HDF4.2r0, HDF5-1.6.3, Szip1.2

• Next release – Oct 2010– Version 5.2.17

• in-sync with SDP TOOLKIT 5.2.17 • with new releases of HDF4 (2.5), 5 (1.8.5)

19

HEG (HDF-EOS2 to GeoTIFF Converter):HEG (HDF-EOS2 to GeoTIFF Converter):

• Last release – Dec 2009– Version 2.10– Patch release Jan 2010

• Next release – Dec 2010– Version 2.11

Software Releases

20

Software Releases

HDF-EOS Plug-in for HDFView:HDF-EOS Plug-in for HDFView:

• Last release – Apr 2010– For HDFView Version 2.6

• Next release – Early 2011– For HDFView Version 2.7 to be released– Version 2.6 plug-in works with HDFView 2.7

21

Software Releases

HECONVERTHECONVERT ((HDF-EOS2 to HDF-EOS5 Converter):

• Last releases – 2001, 2004 No Versioning , Oct. 2008 Version 2.0

• Next release – Nov. 2010 Version 3.0

» Fix for some missing attributes in the converted file

DumpMetadataDumpMetadata (ASCII Dump of HDF-EOS metadata)

• Last release – Dec. 2008 Version 1.0

22

TOOLKIT:TOOLKIT:

– XML metadata writing capability for inventory metadata in SDP toolkit (5.2.16 release).

Added XML style sheet to new TOOLKIT distribution for metadata display with Internet Explorer and other browsers

– Fix for writing metadata into an HDF file when multiple HDF files are open

– libiconv delivery in tar files for Mac and Cygwin – Fix for setting correct endianness flag in linux for eph tools

(Fix is for MOPITT eph files reading)

Recent Major Development/Fixes

23

HDF-EOS2:HDF-EOS2:

– Added Dimension scales for Swath and Grid

Fix for:– Subsetting with GDdefboxregion results in an

extra row and/or column in linux (round-off problem)

– Inconsistency in dimensions between real dims and the values in the structure metadata

– Problem on setting other compressions for swath and grid when disabling szip compression

Recent Major Development/Fixes

24

– Segfault in creating swath when HDF files opened in read-only mode

– Segfault in EHinquire function because of a VGroup with class "CDF0.0”

– Segfault in GDapi (sincos redefinition problem) – Segfault in sample test grid subsetter

– compinfo functions must return -1 only for errors (not for COMP_NONE)

– GDrs2ll should fail for any projection other than BCEA

Recent Major Development/Fixes

25

HDF-EOS5:HDF-EOS5:

– Added Dimension scales to Swath, Grid, and Zonal Average fields

– Fix for problem in setting fillvalue; Fillvalue is only for the field defined right after

setting fillvalue

Recent Major Development/Fixes

26

HDF-EOS Plug-in for HDFView:HDF-EOS Plug-in for HDFView:

– Support for all features in HDFView2.6

– Correction for displayed lat/lon in a swath around the international date line

– Bug fix for HDF5 fillvalue and chunking info display

– Bug fixes for saving HDF-EOS5 files after modification

Recent Major Development/Fixes

27

HEG: HEG:

Added Product Support:Added Product Support:– AMSR-E Daily 25 Km Sea Ice AE_SI25 – AMSR-E Daily land AE_Land3 – MOD29E1D: MODIS/Terra Sea Ice Extent & IST

Daily L3 Global 4km EASE-Grid Day, Version 4 – Tested many new products and updated the

list of Hegable MODIS, ASTER, AIRS, MISR, and AMSR-E products in HEG Users Guide and HEG web site

Recent Major Development/Fixes

28

Enhancement:Enhancement:

– Added a new projection, Cylindrical Equa Area (CEA), to support AMSR-E Daily Land product.

– Pixel resampling support for gridded data with No-Reprojection

– Outputting all fields in a file to a single HDF-EOS output when HDF-EOS output is requested.

– Setting of projection parameters for multiband geotiff only once in the GUI

Recent Major Development/Fixes

29

Bug fixes:Bug fixes:

– Pixel size unit correction for consistency between conversion and stitch GUIs.

– Problem resolution with several products MOD43C3, MOD10CM, MOD29P1N, AE_RnGd, AE_Rain

– Correcting sphere radius written to outputs for MODIS SIN grids

– Resolving problems • with band selection in stitch GUI • with occasional core dump (runs that take long

time to complete) in Datapool HEGserver

Recent Major Development/Fixes

30

Future Plans

TOOLKIT/HDF-EOS:TOOLKIT/HDF-EOS:

• XML Metadata reading in TOOLKIT• Support for new versions of HDF4 - HDF5• Update of DEM Tools for land-sea mask and new 500m

DEM data• Fix for getting correct compression info for a field in

HDF-EOS5, when GDcompinfo/SWcompinfo fails to get the info from Structure metadata

• Support (or improvements) for 64-bit platforms• Fix for Wrong values for some indices returned from

SWindexinfo() in both hdfeos2 and hdfeos5

31

Future Plans

• Store info in structural metadata somewhere else

• Auto Configuration for TOOLKIT• RPMs for HDF-EOS and Toolkit• Validation tool for HDF-EOS5 files• Fix for user/DAAC reported bugs

32

Future Plans (cont.)

HDF-EOS Plug-in for HDFView:HDF-EOS Plug-in for HDFView:• More functionalities :

– Cut/paste objects, modify/delete attributes.– Create new grid/swath

– Support Zonal Average object for HDF-EOS5– Add World Coast Line map to HDFView Plug-in

HEG:HEG:– Performance improvement for Lambert Azimuthal

Projection conversion – Support HEG in MAC Intel– Resolve reported bugs or requests from DAACs/Users

33

Availability

• Access to libraries, applications, Documentation:

TOOLKITTOOLKIT:: http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html HDFViewHDFView::http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/http://newsroom.gsfc.nasa.gov/sdptoolkit/HDFView/

HDFView_hdfeos_plugin.htmlHDFView_hdfeos_plugin.htmlHEGHEG: : http://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.htmlhttp://newsroom.gsfc.nasa.gov/sdptoolkit/HEG/HEGHome.html

HDF-EOSHDF-EOS:: http://www.hdfeos.org/http://www.hdfeos.org/ ( new( new look, sample codes + look, sample codes + HDF-EOS Forum HDF-EOS Forum ))

• Email– Abe_Taaheri@raytheon.com– Landover_PGSTLKIT@raytheon.com

34

Thank You!