tools to improve the usability of nasa hdf data
TRANSCRIPT
![Page 1: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/1.jpg)
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 1
Tools to Improve the Usability of NASA HDF Data
Kent Yang and Joe LeeThe HDF Group
April 17, 2012
![Page 2: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/2.jpg)
www.hdfgroup.org
Tutorial Outline
Introduction
5 Tools
1. H4CF Conversion Toolkit (h4cf)
2. h4toh5 Conversion Tool (h4toh5)
3. HDF-EOS5 to netCDF-4 Converter (eos52nc4)
4. HDF-EOS5 Augmentation Tool (aug_eos5)
5. HDF-EOS2 Dumper (eos2dump)
Demo
HDF/HDF-EOS Workshop XV 2April 17, 2012
![Page 3: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/3.jpg)
www.hdfgroup.org
Who Said Users Don’t Matter?
April 17, 2012 HDF/HDF-EOS Workshop XV 3
![Page 4: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/4.jpg)
www.hdfgroup.org
NASAHDF/HDF-EOS
Data
Usability of NASA HDF Data
April 17, 2012 HDF/HDF-EOS Workshop XV 4
IDL
MATLAB
Panoply IDV
A school teacher’s viewPanoply IDV
A school teacher’s expectation
![Page 5: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/5.jpg)
www.hdfgroup.org
The 5 Tools
April 17, 2012 HDF/HDF-EOS Workshop XV 5
Improve the usability of NASA Data
Supported by The HDF Group
Funded by NASA / Raytheon
http://hdfeos.org/software/tool.php
![Page 6: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/6.jpg)
www.hdfgroup.org
h4cf
eos2 dump
aug_eos5
eos52nc4
h4toh5
hdf4
5 Tools that Improve Usability
April 17, 2012 HDF/HDF-EOS Workshop XV 6
eos2
eos5ASCII
NETCDF-4
NETCDF-3Panoply / IDV
MATLAB IDLNCL
![Page 7: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/7.jpg)
www.hdfgroup.org
H4CF Conversion Toolkit
Follow CF conventionBoth Library and Toolshttp://hdfeos.org/software/h4cflib
April 17, 2012 HDF/HDF-EOS Workshop XV 7
hdf4
eos2
NETCDF-4
NETCDF-3 h4cfPanoply / IDV
![Page 8: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/8.jpg)
www.hdfgroup.org
H4CF Conversion Toolkit
Two ways to build:
1. Use configure script.
2. Edit Makefile
template.
April 17, 2012 HDF/HDF-EOS Workshop XV 8
![Page 9: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/9.jpg)
www.hdfgroup.org
H4CF Conversion Library
• Variables (datasets) list• Attributes (metadata) list
• All Vars & Attrs follow CF conventions• Develop your own CF application!
April 17, 2012 HDF/HDF-EOS Workshop XV 9
![Page 10: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/10.jpg)
www.hdfgroup.org
H4CF Library APIs Example
// open the example HDF fileh4cf_open("geo.hdf");
// HDF file attributes are obtained in a C++ STL map objectconst map<string, string> file_attrs = h4cf_get_file_attrs();
// HDF file variables are obtained in a C++ STL list objectconst list<var*> pvars = h4cf_get_vars();
// close the HDF fileh4cf_close();
April 17, 2012 HDF/HDF-EOS Workshop XV 10
![Page 11: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/11.jpg)
www.hdfgroup.org
H4CF Conversion Tools
2 Tools based on H4CF APIs:
cdldumper
h4tonccf
April 17, 2012 HDF/HDF-EOS Workshop XV 11
![Page 12: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/12.jpg)
www.hdfgroup.org
cdldumper
• Usage: cdldumper filename.hdf
• Dump file in Common Data Language.• Useful for quick check and debugging purpose.
April 17, 2012 HDF/HDF-EOS Workshop XV 12
![Page 13: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/13.jpg)
www.hdfgroup.org
cdldumper Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 13
Number of attributes=1_FillValue=255
![Page 14: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/14.jpg)
www.hdfgroup.org
cdldumper Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 14
%./cdldumper -H AMSR_E_L3_5DaySnow_V09_20050126.hdf netCDF AMSR_E_L3_5DaySnow_V09_20050126.hdf {dimensions: XDim_Northern_Hemisphere = 721, XDim_Southern_…variables: ubyte Flags_NorthernPentad(YDim_Northern_Hemis… Flags_NorthernPentad:_FillValue = 255; Flags_NorthernPentad:origname = "/Northern… Flags_NorthernPentad:long_name = "/Northern… Flags_NorthernPentad:coordinates = "latitude…
Number of attributes=4_FillValue=255orig_namelong_namecoordinates
![Page 15: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/15.jpg)
www.hdfgroup.org
h4tonccf
• Usage: h4tonccf filename.hdf [filename.nc]• generates either netcdf-3 or netcdf-4 file.
• follows netcdf classic model.
April 17, 2012 HDF/HDF-EOS Workshop XV 15
![Page 16: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/16.jpg)
www.hdfgroup.org
h4tonccf Demo
April 17, 2012 HDF/HDF-EOS Workshop XV 16
![Page 17: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/17.jpg)
www.hdfgroup.org
Don’t Follow CF Conventions Blindly
April 17, 2012 HDF/HDF-EOS Workshop XV 17
[1] CF: scale * value + offset[2] MODIS: scale * (value – offset)
Scale = 0.01Value = 5000Offset = -15000.0
[1] 0.01 * 5000 + (-15000) = -14950[2] 0.01 * (5000 – (-15000)) = 200
![Page 18: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/18.jpg)
www.hdfgroup.org
-14950 Kelvin?
April 17, 2012 HDF/HDF-EOS Workshop XV 18
![Page 19: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/19.jpg)
www.hdfgroup.org
MODIS Scale / Offset Correction
HDF/HDF-EOS Workshop XV 19April 18, 2012
![Page 20: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/20.jpg)
www.hdfgroup.org
Converted File Size Comparison
April 17, 2012 HDF/HDF-EOS Workshop XV 20
72M
128M
656M
HDF-EOS2
Netcdf-4
Netcdf-3
9X
![Page 21: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/21.jpg)
www.hdfgroup.org
Limitation – No Support Yet
April 17, 2012 HDF/HDF-EOS Workshop XV 21
MISR (SOM/LAMAZ)
MERRAAVHRRLISTO
QuikSCAT/SeaWindMOPITT L2
etc.
![Page 22: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/22.jpg)
www.hdfgroup.org
h4toh5 Conversion Tool
Convert any hdf4 to hdf5.
http://hdfgroup.org/h4toh5
April 17, 2012 HDF/HDF-EOS Workshop XV 22
eos2
NETCDF-4
h4toh5
Usage: h4toh5 –eos –nc4 in.hdf out.nc
Static Binary w/ –eos & -nc4:
http://hdfeos.org/software/h4toh5/bin/
![Page 23: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/23.jpg)
www.hdfgroup.org
h4tonccf vs. h4toh5
April 17, 2012 HDF/HDF-EOS Workshop XV 23
h4tonccf h4toh5w/ -eos –nc4
Handle non-eos2 hdf4
Yes No
FollowCF
conventions
Yes No
Outputnetcdf classic
Yes No
Outputnetcdf
enhanced
No Yes
![Page 24: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/24.jpg)
www.hdfgroup.org
eos52nc4
• converts hdf-eos5 to netcdf-4.• follows netcdf enhanced model.• does not follow CF conventions. • Usage: eos52nc4 test.he5 test.nc• http://hdfeos.org/software/convert_hdfeos5.php
April 17, 2012 HDF/HDF-EOS Workshop XV 24
eos5
NETCDF-4eos52nc4
![Page 25: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/25.jpg)
www.hdfgroup.org
aug_eos5
• augments hdf-eos5 to netcdf-4.• Both HDF-EOS5 and NetCDF4 can access it.• Usage: aug_eos5 [-i|-f map] test.h5
• modifies the input file directly. • does not follow CF conventions.• http://hdfeos.org/software/aug_hdfeos5.php
April 17, 2012 HDF/HDF-EOS Workshop XV 25
eos5
NETCDF-4aug_eos
5
![Page 26: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/26.jpg)
www.hdfgroup.org
hdf5
netcdf4
Augmentation Concept
April 17, 2012 HDF/HDF-EOS Workshop XV 26
hdf-eos5HDF-EOS5
APINETCDF-4
API
aug_eos5
augmentation
hdf-eos5
HDF5 API
![Page 27: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/27.jpg)
www.hdfgroup.org
Why NetCDF-4 Doesn’t Work
April 17, 2012 HDF/HDF-EOS Workshop XV 27
[nLevels][nTimes]
NETCDF-4 API
hdf-eos5
![Page 28: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/28.jpg)
www.hdfgroup.org
What aug_eos5 Does
April 17, 2012 HDF/HDF-EOS Workshop XV 28
[nLevels][nTimes]
NETCDF-4 API
hdf-eos5
![Page 29: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/29.jpg)
www.hdfgroup.org
Options in aug_eos5
1. Default option
• -i option
2.-f map_file option
April 17, 2012 HDF/HDF-EOS Workshop XV 29
![Page 30: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/30.jpg)
www.hdfgroup.org
Default option in aug_eos5
Grid
•XDim, YDim: calculated
• Other dims: default fill value
Swath / ZA• All dims: default fill value
April 17, 2012 HDF/HDF-EOS Workshop XV 30
![Page 31: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/31.jpg)
www.hdfgroup.org
Better -i option in aug_eos5
• Grid• XDim, YDim: calculated
•Other dims: 1,2,…,n
• Swath / ZA
•All dims: 1,2,…,n
April 17, 2012 HDF/HDF-EOS Workshop XV 31
![Page 32: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/32.jpg)
www.hdfgroup.org
Best -f option in aug_eos5
• Needs an input map file.
• Input file specifies 3 rules:1. What to be filled in
2. How to fill in
3. Where to get the values• Most useful and flexible option
April 17, 2012 HDF/HDF-EOS Workshop XV 32
![Page 33: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/33.jpg)
www.hdfgroup.org
aug_eos5 Input File Format
Each line in input file contains
1. How: 0,1,2
2. What: dim name3. Where: data file or existing variable in file.
4. (Optional) Multiple grid/swath/za case only:• grid/swath/za name:<name>
April 17, 2012 HDF/HDF-EOS Workshop XV 33
![Page 34: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/34.jpg)
www.hdfgroup.org
aug_eos5 Input File Example
0 nTimes Time
1 nLevels data.txt
2 nCandidates
April 17, 2012 HDF/HDF-EOS Workshop XV 34
Copy the data
Read file data.txt
Fill 1,2,…, n
How What Where
95% cases – Copy the data
![Page 35: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/35.jpg)
www.hdfgroup.org
Multiple Swath Case
swath name: BrO
0 nTimes Time
1 nLevels data1.txt
2 nCandidates
April 17, 2012 HDF/HDF-EOS Workshop XV 35
How What Where
swath name: BrO column
2 nTimes
1 nLevels data2.txt
Multiple
![Page 36: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/36.jpg)
www.hdfgroup.org
aug_eos5 vs eos52nc4
April 17, 2012 HDF/HDF-EOS Workshop XV 36
aug_eos5 eos52nc4
hdfeos-5 Yes No
netcdf-4 Yes Yes
![Page 37: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/37.jpg)
www.hdfgroup.org
HIRDLS/MLS Team Use aug_eos5
April 17, 2012 HDF/HDF-EOS Workshop XV 37
So You Can!
![Page 38: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/38.jpg)
www.hdfgroup.org
eos2 dump
HDF-EOS2 Dumper Tool
Useful for tools that can’t handle some projections.
April 17, 2012 HDF/HDF-EOS Workshop XV 38
eos2
ASCII
MATLAB IDLNCL
http://hdfeos.org/zoo
![Page 39: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/39.jpg)
www.hdfgroup.org
Important Tool for Zoo Maintenance
April 17, 2012 HDF/HDF-EOS Workshop XV 39
![Page 40: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/40.jpg)
www.hdfgroup.org
Why Tools Need HDF-OES2 Dumper
April 17, 2012 HDF/HDF-EOS Workshop XV 40
Lat / Lonin
ASCII
MATLAB IDLNCL
HDF-EOS2dumper
LaRC MISR SOM Projection
HDF-EOS2
Lat / Lon
![Page 41: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/41.jpg)
www.hdfgroup.org
HDF-EOS2 Dumper Options
Usage: eos2dump options filename.hdf
What: 1=lat / 2 =lonFormat: m=5 values per line
April 17, 2012 HDF/HDF-EOS Workshop XV 41
![Page 42: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/42.jpg)
www.hdfgroup.org
eos2dump Latitude Example
eos2dump –c1 filename.hdf > latitude.txt
What: latFormat: one value per line (default)
April 17, 2012 HDF/HDF-EOS Workshop XV 42
9080706050403020100-10-20-30
![Page 43: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/43.jpg)
www.hdfgroup.org
eos2dump Longitude Example
eos2dump –c2m filename.hdf > longitude.txt
What: lon
Format: 5 values per lineMATLAB cannot read it.
April 17, 2012 HDF/HDF-EOS Workshop XV 43
180 175 170 165 160155 150 145 140 135130 125 120 115 110
![Page 44: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/44.jpg)
www.hdfgroup.org
Got Tools?
April 17, 2012 HDF/HDF-EOS Workshop XV 44
![Page 45: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/45.jpg)
www.hdfgroup.org
h4cf
eos2 dump
aug_eos5
eos52nc4
h4toh5
hdf4
Enjoy the NASA HDF Data!
April 17, 2012 HDF/HDF-EOS Workshop XV 45
eos2
eos5ASCII
NETCDF-4
NETCDF-3Panoply / IDV
MATLAB IDLNCL
NASA HDFdata users
![Page 46: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/46.jpg)
www.hdfgroup.org
… and observe the Earth!
April 17, 2012 HDF/HDF-EOS Workshop XV 46
![Page 47: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/47.jpg)
www.hdfgroup.org
The HDF Group
HDF/HDF-EOS Workshop XV 47
Thank you !
April 17, 2012
![Page 48: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/48.jpg)
www.hdfgroup.org
Acknowledgements
HDF/HDF-EOS Workshop XV 48April 17, 2012
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, orrecommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
![Page 49: Tools to improve the usability of NASA HDF Data](https://reader036.vdocuments.mx/reader036/viewer/2022081422/55639504d8b42a7c458b4f69/html5/thumbnails/49.jpg)
www.hdfgroup.org
The HDF Group
Questions/comments?
HDF/HDF-EOS Workshop XV 49April 17, 2012