hdf augmentation: interoperability in the last mile
TRANSCRIPT
www.hdfgroup.org
The HDF Group
AGU Fall Meeting
HDF Augmentation: Interoperability in the Last MileJoel Plutchak, Michael J Folk, Ted Habermann and Larry Knox
The HDF Group
1December 17, 2014
Paper IN32A-06
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Repurposing
DataCollection
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
Analysis, Visualization and Publication Tools
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
Analysis, Visualization and Publication Tools
Analysis, Visualization and Publication Tools
Catalogs, Data Discovery,
Preservation
www.hdfgroup.orgAGU Fall Meeting
Data Life Cycle – First and Last Miles
Archive Discovery AnalysisQuestion Processing Distribution
Principal Investigator Someone Else
Repurposing
DataCollection
Analysis, Visualization and Publication Tools
Analysis, Visualization and Publication Tools
Catalogs, Data Discovery,
Preservation
www.hdfgroup.orgAGU Fall Meeting
Why Augment?
July 8 – 11, 2014 8
• Serve intended purpose better.
• Serve unexpected new users.
• Serve unexpected new purpose.
• Answer unexpected new questions.
• Data available to multiple tools and applications.
• More “standard” and more widely accessible.
• Cost effective information preservation.
www.hdfgroup.orgAGU Fall Meeting
How?
July 8 – 11, 2014 9
• Add extra information: • Metadata that enhances discoverability, usability, archivability
• Geolocation information
• Rename or move objects around• Move elements from within a hierarchy to the top level
• Rename elements that clash with conventions
• Reduce complexity of organization• Flatten hierarchical organization or group structures
• Hide objects not commonly understood• e.g., data reference types
While preserving original purpose and tools (if possible)
www.hdfgroup.orgAGU Fall Meeting
Archive
netCDF4 access to HDF-EOS5 data
Archive
Discovery AnalysisQuestion Processing DistributionDataCollection
HDF5 Data
HDF-EOS5Metadata
Principal Investigator Someone Else
Principal Investigator uses HDF-EOS5 tools for analysis of
HDF5 data
www.hdfgroup.orgAGU Fall Meeting
netCDF4 metadata added to HDF-EOS file
Archive
netCDF4 access to HDF-EOS5 data
Archive
Discovery AnalysisQuestion Processing DistributionDataCollection
Future User uses netCDF-4 tools for analysis of
HDF5 data
HDF5 Data
HDF-EOS5Metadata
HDF5 Data
netCDF4Metadata
Principal Investigator Someone Else
Principal Investigator uses HDF-EOS5 tools for analysis of
HDF5 data
www.hdfgroup.orgAGU Fall Meeting
netCDF4 metadata added to HDF-EOS file
HDF5 data can be used with all tools
Archive
netCDF4 access to HDF-EOS5 data
Archive
Discovery AnalysisQuestion Processing DistributionDataCollection
Future User uses netCDF-4 tools for analysis of
HDF5 data
HDF5 Data
HDF-EOS5Metadata
netCDF4Metadata
Principal Investigator Someone Else
Principal Investigator uses HDF-EOS5 tools for analysis of
HDF5 data
www.hdfgroup.orgAGU Fall Meeting
H5augjpss flattens and hides metadata
Archive
JPSSMetadata
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
www.hdfgroup.orgAGU Fall Meeting
H5augjpss flattens and hides metadata
Archive
JPSSMetadata
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
The Integrated Data Viewer (IDV) supports netCDF4 Classic
(no groups) and needs conventional metadata
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
JPSS Metadata(some hidden)
HDF5 Data
?
www.hdfgroup.orgAGU Fall Meeting
H5augjpss flattens and hides metadata
Archive
JPSSMetadata
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
The Integrated Data Viewer (IDV) supports netCDF4 Classic
(no groups) and needs conventional metadata
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
JPSS Metadata(some hidden)
HDF5 Data
H5augjpss
Two files, one for each access tool.
?Flat CF Metadata
www.hdfgroup.orgAGU Fall Meeting
JPSS NcML – Non-invasive Approach
Archive DiscoveryAnalysis
JPSSMetadata
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
JPSSMetadata
HDF5 Data
?
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
The Integrated Data Viewer (IDV) supports netCDF4 Classic
(no groups) and needs conventional metadata
www.hdfgroup.orgAGU Fall Meeting
JPSS NcML – Non-invasive Approach
Archive DiscoveryAnalysis
JPSSMetadata
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
JPSSMetadata
HDF5 Data
?CF Metadata in
External NcML File
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
The Integrated Data Viewer (IDV) supports netCDF4 Classic
(no groups) and needs conventional metadata
www.hdfgroup.orgAGU Fall Meeting
JPSS NcML – Non-invasive Approach
Archive DiscoveryAnalysis
JPSSMetadata
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF5 Data
JPSSLibrary
JPSSMetadata
HDF5 Data
JPSS Library continues to work for current users
?CF Metadata in
External NcML File
JPSS HDF5 files use structures (groups, region references) that are not supported by older tools and lack conventional metadata.
The Integrated Data Viewer (IDV) supports netCDF4 Classic
(no groups) and needs conventional metadata
www.hdfgroup.orgAGU Fall Meeting
HDF4 Maps in XML Enable Diverse Access
Archive DiscoveryAnalysis
HDF-EOS2Metadata
Current users use HDF-EOS2 library and tools for analysis of HDF4 data
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF4 Data
Analysis, Visualization and Publication Tools
HDF-EOS2Library
www.hdfgroup.orgAGU Fall Meeting
HDF4 Maps in XML Enable Diverse Access
Archive DiscoveryAnalysis
HDF-EOS2Metadata
Current users use HDF-EOS2 library and tools for analysis of HDF4 data
Future users may not have the HDF-EOS2 library
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF4 Data
Analysis, Visualization and Publication Tools
HDF-EOS2Library
FutureAnalysis, Visualization and
Publication Tools
HDF-EOS4Metadata
HDF4 Data
?
www.hdfgroup.orgAGU Fall Meeting
HDF4 Maps in XML Enable Diverse Access
Archive DiscoveryAnalysis
HDF-EOS2Metadata
Current users use HDF-EOS2 library and tools for analysis of HDF4 data
Future users may not have the HDF-EOS2 library
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF4 Data
Analysis, Visualization and Publication Tools
HDF-EOS2Library
FutureAnalysis, Visualization and
Publication Tools
HDF-EOS4Metadata
HDF4 Data
HDF4 Map contains metadata
in XML
?HDF4 MAP (XML)
www.hdfgroup.orgAGU Fall Meeting
HDF4 Maps in XML Enable Diverse Access
Archive DiscoveryAnalysis
HDF-EOS2Metadata
Current users use HDF-EOS2 library and tools for analysis of HDF4 data
Future users may not have the HDF-EOS2 library
Archive Discovery AnalysisQuestion Processing DistributionDataCollection
Principal Investigator Someone Else
HDF4 Data
Analysis, Visualization and Publication Tools
HDF-EOS2Library
FutureAnalysis, Visualization and
Publication Tools
HDF-EOS4Metadata
HDF4 Data
HDF4 Library continues to work for current users
?HDF4 MAP (XML)
www.hdfgroup.orgAGU Fall Meeting
Conclusion
December 17, 2014 23
Augmentation enables efficient data re-use
Several approaches can be used to augment data and metadata
Invasive approaches result in new files with different metadata
Non-invasive approaches preserve current use
The HDF Group and others provide tools and methods
www.hdfgroup.orgAGU Fall Meeting
Thank you!
Questions?
December 17, 2014 24
This work was supported by subcontract 114820 under prime contract NNG10HP02C, funded by the National
Aeronautics and Space Administration (NASA). Any opinions, findings, or conclusions expressed in this material
are those of the author and do not necessarily reflect the views of NASA.