opendap services for esg september 25, 2014
DESCRIPTION
Performance Measures x.x, x.x, and x.x. OPeNDAP Services for ESG September 25, 2014. Peter Fox, Patrick West, Stephan Zednik RPI. In ESG II (in regard to data). Server side aggregation DAP object transfer via HTTP and GridFTP GSIFTP integration, myProxy support - PowerPoint PPT PresentationTRANSCRIPT
1
Earth System Grid Center for Enabling Technologies
OPeNDAP Services for ESG
April 21, 2023
Peter Fox, Patrick West, Stephan ZednikRPI
Performance Measures x.x, x.x, and x.x
2
Earth System Grid Center for Enabling Technologies: (ESG-CET)
In ESG II (in regard to data)
Server side aggregation DAP object transfer via HTTP and GridFTP GSIFTP integration, myProxy support OPeNDAP-g -> led to architecture of OPeNDAP/ Hyrax
(primarily BES is from OPeNDAP-g)
3
Earth System Grid Center for Enabling Technologies: (ESG-CET)
Requirements leading to OPeNDAP-g
Separation of the core Data Access Protocol (DAP) from the transport protocol (HTTP).
High Performance Computing. The previous CGI based servers did not have the capacity required by ESG. Error and memory handling added.
Security. Once the OPeNDAP was independent of the transport protocol, adding security was possible by relying on the Globus gsiFTP system.
Aggregation. OPeNDAP 3.0 did not operate on aggregated datasets. OPeNDAP-g does.
Transport protocol independence and HPC were incorporated back into OPeNDAP leading to the current version. Security and aggregation remain an ESG only feature.
4
Earth System Grid Center for Enabling Technologies: (ESG-CET)
OPeNDAP-g Architecture
Dispatcher BES
OPeNDAP-g Dispatcher (e.g. ESG Front-end Server) Receives requests and asks the BES to fill them Uses Apache Modules Does not directly ‘touch’ data, handles URLs
Data
Back End Server (BES) Reads data files, Databases, et c., returns info May return DAP objects or other data (netCDF)
55
OPeNDAP Hyrax Architecture
OLFS BES
OPeNDAP Lightweight Front end Server (OLFS) Receives requests and asks the BES to fill them Uses Java Servlets Does not directly ‘touch’ data Multi-protocol
Data
Back End Server (BES) Reads data files, Databases, et c., returns info May return DAP2 objects or other data Does not require web server
Client
66
GridFTPDAP2
GridFTPDAP2
HTTPDAP2HTTPDAP2
ASCII outputASCII output
HTML formHTML form
Info outputInfo output
OPeNDAP Lightweight Front end ServerOPeNDAP Lightweight Front end Server
THREDDS**THREDDS**
Request Formulation**Request Formulation**
Req
ues
t fr
om
cli
ent
Res
po
nse
to
cli
ent
BE
S
SOAP-DAP (HTTP)
DAP2 (GridFTP, HTTP)
RDF, OWL, JSON (HTTP)
PML outputPML output
77
Hyrax/ Back-end Server
Network Protocol andProcess start/stopactivities
Data Store Interfacese.g. IOSP
BES Framework
PPT/PPTS*Initialization/Termination
DAP2Access
NetCDF3 HDF4 RDF/ SPARQL…
Provenance
Commands**e.g. server-side, Ferrett
BES Commands/ XML Documents
*PPT is built in (other protocols)**Some commands are built in
Data DataData
DataCatalogs
8
Earth System Grid Center for Enabling Technologies: (ESG-CET)
OPeNDAP-g services for ESG
Data access via Dataportals. In this Use Case users interact with the portal, browse the catalogs and decide what data to download. The portal passes the request to OPeNDAP-g which executes it and returns the data to the portal. The portal returns a URL to the user to download the data. Data access via netCDF library. In this Use Case users link their applications with OPeNDAP-g client library for netCDF. The user can “open” a URL that refers to ESG data. The library fetches the data for the user and makes it a local netCDF file The performance requirements for ESG II were met. However, ESG-CET scales up these requirements.
9
Earth System Grid Center for Enabling Technologies: (ESG-CET)
Status of the Community OPeNDAP Server Software
Together the OLFS and the BES are known as Hyrax Hyrax 1.6 provides support for NcML-based
aggregation Data response streamed back as netCDF file RDF response type Updated DDX response type (Data Definition XML) Beginning development of DataDDX – multi-mime
response with data and DDX Full security audit and static code analysis certification
to comply with NOAA and NASA requirements
10
Earth System Grid Center for Enabling Technologies: (ESG-CET)
ESG-CET and data
Large data sets, numbers and sizes• High performance• Flexible architecture, both client and several types and
numbers of servers• Aggregation• Server side operations• Multiple transport protocol options
Full ESG security support as well as loose federation Read-only client access via API (netCDF/CDM) To satisfy the new goals, the OPeNDAP services for ESG have
been re-architected. We now use parts of the standard OPeNDAP framework Hyrax,
focusing on high performance for the client side and extended flexibility.
11
ESG-CET and Products (server side functions)
Goal: drop in replacement for the TDS part of FTDS in LAS
Requires netCDF-Java Input-Output Service Provider (IOSP) adapter for Hyrax/BES
Use case examination will be required
Earth System Grid Center for Enabling Technologies: (ESG-CET)
12
Security Infrastructure status
OPeNDAP BES Security• SSL authentication between gridFTP middle tier and the
BES. No persistent SSL connections are maintained RNI Integration with ESG Security Infrastructure
• RNI client supports gsiFTP connections to ESG GridFTP servers
• ESG GridFTP server handles authentication of user• Neill's ESG GridFTp authz callout plugin handles
authorization of data request• ESG/RNI GridFTP DSI module handles data request,
forwards request to ESG BES server running RNI module
Earth System Grid Center for Enabling Technologies: (ESG-CET)
13
Client status
RNI Version 0.1 implemented using netCDF version 3 and OPeNDAP’s libnc-dap
In communication with Unidata regarding integration of RNI client with new NetCDF version 4
Developing ncml aggregation in both client and server RNI
Full ESG security support
Earth System Grid Center for Enabling Technologies: (ESG-CET)
14
Earth System Grid Center for Enabling Technologies: (ESG-CET)
The Remote NetCDF Invocation (RNI)
The client is the netCDF library. It has exactly the same API as the standard C library netCDF, but it can deal with local files or files reachable via HTTP, PPT or gridFTP. The third tier, the BES server can be reached only via PPT. NetCDF services for all NetCDF calls are implemented a a BES module. The middle tier, acts like a proxy between the RNI client and server and deals with security.
15
Earth System Grid Center for Enabling Technologies: (ESG-CET)
RNI Architecture
CLIENT DATA
GridFTPOPeNDAP
BES
NetCDFLibrary
RNI Module
connection acts like
RNI Library
16
Earth System Grid Center for Enabling Technologies: (ESG-CET)
Characteristics of the RNI as part of a data access system
Full Support of standard OPeNDAP URLs. RNI is being developed with the integrated Unidata/OPeNDAP netCDF library (and CDM)
Transparent access to either standard netCDF files and aggregated datasets via the NetCDF Markup Language (NCML).
For remote containers, all write operations are disable for security. That is, for HTTP/HTTPS, PPT and gridFTP/gsiFTP the RNI system is a read only API.
RNI utilizes Just in Time access. Caching is only for metadata. No pre-fetching of data.
RNI transparently accesses secure (gsiFTP, HTTPS) or insecure (gridFTP, HTTP) remote data.
17
Earth System Grid Center for Enabling Technologies: (ESG-CET)
Status of the RPI work (OPeNDAP)
The primary accomplishments for this subproject in the past year has been• The complete request-response for all netCDF API calls has been
completed, that is, the two ends have been developed.• We have highly optimized core components of the standard
OPeNDAP framework to support the performance goals.• We have established the knowledge of how the middle tier (the
proxy) will be incorporated into the complete system.• Our work was presented in AGU fall 2007 and EGU 2008.
So far all of the goals established for the first stage of the project has been completed.
Next stage• integration with product server for gateway and data node data
access• Functional (with enhancements) replacement for TDS
18
Future, will not elaborate
Storage Resource Manager / DMLite as a client Return as RDF Return as PML (Provenance – Proof Markup Language)
Earth System Grid Center for Enabling Technologies: (ESG-CET)