OpenDAP Server-side Functions for Multi-Instrument Aggregation

ESIP Session: Advancing the Power and Utility of Server-side Aggregation

Jon C. Currey (NASA), Aron Bartle (Mechdyne),James Gallagher (OpenDAP)

Challenge

• How to match and merge high resolution observation data (L1 & L2) from instruments on separate spacecraft?

• How do I get just the data I need? I don’t want it all!• How do I handle different footprint sizes?• What about spectral differences?• Maybe I should just stick with gridded data

Science Use Cases

• There are many studies that require access to full resolution observation data. Today we will address only a few:– Intercalibration (GSICS community)– Validation of L2 Retrievals – Surface site data mining

Multi-Instrument Intercalibration (MIIC)

• Funded by ROSES ACCESS• Spinoff of the CLARREO mission • Addresses how to efficiently match and access

instrument data from multiple data centers

MIIC Tiered Architecture

How does MIIC Aggregate Data?

• Two critical pieces:– Event prediction– OPeNDAP server-side functions– Together they perform efficient aggregation!

Part 1 – MIIC Event Prediction Process

• LEO-LEO intercalibration event opportunity based on the time when the primary instrument “P” FOVs are within the tent structure built using the secondary instrument location at A and fictitious locations at A+ and A- based on allowable observation time differences (eg., 2.5 minutes). Table shows the effect of orbit altitude differences on frequency of events (crossings).

• Event boundary trimmed to where instrument viewing condition differences within specified ranges (use viewing zenith, relative azimuth, and solar zenith angles)

Part 1 - MIIC Event Prediction Sample Results Can find event opportunities for many spacecraft combinations!

NPP VIIRS vs. Aqua CERES Event Prediction, Oct. 2013, daytime, SZA < 75°, |Δ VZA| < 10°, |Δ RAZ| < 30°, |Δ time| < 2.5 min.

Part 2 – OPeNDAP Server-side Functions

• Two main server-side functions: Tuple and Histogram• Use OPeNDAP nested functions• Data variable functions identify parameters to return• Filter functions define filter variables and ranges• Filter variable functions are AND’ed• Tuple function returns full resolution filtered Array data • Histogram function returns filtered data binned in one or two

dimensions; returns average + statistics for each bin• Option to decimate data prior to tuple and histogram • Server-side functions are generic – should work on all netCDF

and HDF L1 and L2 files

OPeNDAP Tuple Server-side FunctionReturns Full Resolution Data for Single LEO-GEO Predicted Event

Full resolution LEO-GEO predicted matched event, NPP VIIRS vs. GOES13 for Jan. 1, 2013, 51.2S-46.4N, 114.7W-53.2W, T18:49:40Z-T19:17:00Z; 1 km GOES13 Band 1 visible data (left); matched 2km VIIRS Radiance_I3_Avg data from multiple aggregated NPP_VIMD_SS files (right). “Bow tie” fill values at scan angles greater than 31.7° and 56.3° need to be filtered. Accessing GOES and VIIRS data from OPeNDAP servers using tuple server-side function. Full resolution LEO-GEO tuple data is still a significant amount of data to download. Use 2Dhistogram function to grid and reduce data by several orders of magnitude!

OPeNDAP 2DHistogram Server-side Function Returns 0.5° Gridded Data for Single LEO-GEO Predicted Event

One gridded LEO-GEO event, 0.5 degree, NPP VIIRS (right) vs. GOES13 (left) for Jan. 1, 2013. MIIC 2DHistogram server-side function reduces data by a factor of 3500.

Surface Site Data Mining Using MIIC Server-side Filtering and Statistics

Deep Convective Cloud (DCC) Filter implemented within OPeNDAP tuple function; cloud top height >10km, cloud optical depth >10, cloud fraction > 99%, latitude band 25°S – 25°N. MIIC event prediction determines file coverage for region, then retrieves only filtered data from OPeNDAP servers. DCC are large cold bright clouds in the tropics (see LW and SW flux).

Challenging Validation Example

Image on left shows CALIPSO groundtrack (blue) and aggregated VIIRS radiance data

Using MIIC to support CrIMSS Validation Study (X. Liu); need to merge CALIPSO, CRIS, ATMS, VIIRS, and CrIMSS data

Current system returns full resolution data from each instrument within event boundary (“coarse” match) using tuple server-side function

Dealing with multiple instruments types (imager, profiler, spectrometer, sounder) all with different footprint sizes!

Need to develop Spatial Registration Server-side function to match data at the footprint scale!

New Generalized Spectral Convolution Server-side Function

• Needed for intercalibration – convolve imager RSRs with reference spectra• Demonstrated using hyperspectral SCIAMACHY and L1 MODIS data• Allows users to compare (aggregate) imager bands to a hyperspectral reference• Massive reduction in data transfer

Spectral nadir reflectance over deep convective cloud scene from SCIAMACHYdata with MODIS VIS and near-IR bands

More traditional types of aggregation (NCML-like) to be handled in OPeNDAP

• Need to aggregate CALISPO Aerosol and Cloud properties stored in separate files; have same time and spatial scales

• NPP VIIRS SDR channels stored in separate files; need to access geolocation and muliple bands within same read to perform server-side filtering

Summary

• Within the Earth Sciences community there is still a need to provide efficient access to data located across multi-agency data centers

• The MIIC tiered system using OPeNDAP server-side functions is one mechanism to match (aggregate) data from multiple instruments

• Thank you! For more information on MIIC contact jon.c.currey@nasa.gov