grads-dods server an open-source tool for distributed data access and analysis joe wielgosz, brian...

GrADS-DODS ServerAn open-source tool for distributed data access and analysis

Joe Wielgosz, Brian Doty, Jennifer AdamsCOLA/IGES - Calverton, MD

http://www.iges.org/grads/gds

OPeNDAP Working Conference - March 19, 2003

<?xml version="1.0" encoding="ISO-8859-1"?>

<gds name="my data server" home="/"> <catalog temp_size_limit="1000"> <data> <dataset name = "test” file = "testdata/little_endian.ctl“> </data> </catalog>...

GrADS-DODS Server Joe Wielgosz - [email protected] http://www.iges.org/grads/gds

Make GrADS-readable datasets - both gridded and in-situ - accessible across the network, to a diverse range of clients

What the GDS can do

Perform server-side analysis and comparisons against other distributed datasets

servers clients

and more...

web browserLAS

GDS

OPeNDAP server

OPeNDAP server

GDScomparisondata

GRIB

HDF4

netCDF

BUFR

Unidata IDV

ncBrowse

GrADS

Ferret

Matlab

IDL

OPeNDAP

binary

GrADS

Java servlet


What's new since 1.1

New data type support

Station data - GrADS format and BUFR

Remote OPeNDAP data

Subsampling ("striding") for gridded data

Core code refactoredAnagram - generic data server frameworkSwappable, reusable modulesDesigned for efficiency - streaming I/O

XML-based configuration,with more flexibility in:Dataset loadingLoggingSecurityResource management

Improved web interfaceCustom links to help, home, dataset infoURL-based administration interface

Scales better to 1000's of datasetsOrganizes data catalog into directoriesFaster startup and smarter caching


Stable and fast COLA public GDS currently handling >600k hits/month

Administrator-friendly

Complete online documentation: http://www.iges.org/grads/gds/doc

COLA GDS Usage

0

100

200300

400

500

600

700800

900

1000

Sep-01Oct-01Nov-01Dec-01Jan-02Feb-02Mar-02Apr-02May-02Jun-02Jul-02Aug-02Sep-02Oct-02Nov-02Dec-02Jan-03

Thousands

Hits per month

0

100

200

300

400

500

600

700

Unique IPs each month

Total hits unique IPs

<gds> <catalog temp_size_limit="1000"> <data> <dataset name="test" file="testdata/big_endian.ctl" format="ctl" /> <datadir name="/mnt/data1" suffix=".ctl"/> </data> </catalog> <log mode="rotate" frequency="monthly" file="log/gds.log" level="info" /> <grads> <invoker grads_bin="/users/joew/bin/grads"/> </grads> <servlet> <filter-abuse enabled="true" hits="1000" timeout="24" /> <filter-overload enabled="true" limit="20" /> <filter-analysis enabled="true" /> </servlet> <mapper> <service-admin enabled="true" auth="open-sesame" /> </mapper> <privilege_mgr default="public"> <ip_range mask="127.0.0.1" privilege="private" /> <privilege name="public"> <deny path="private_data" /> </privilege> <privilege name="private" /> </privilege_mgr></gds>

Install in minutes (really!)No root privileges neededCross-platform Java and ANSI C

Easy to configureEdit one (simple) XML file, and make updates on-the-fly

SecureRestrict dataset access & resource usage by IP address

And more...Automatic scans for new datasets Detailed loggingGraceful handling of heavy loadsEasily integrated with Apache...


In use, well-received

Positive response from:

• COLA scientists

• GrADS user community - research, corporate, hobbyists

• NOAA/CIRES CDC (earliest adopters outside COLA)

Some public GDS servers: (google on "grads dods server")

COLA Public Data Server: cola8.iges.org:9191 (soon to be wxdata.org)

COLA Monsoon Data Server: monsoondata.org

NOAA/CIRES CDC: www.cdc.noaa.gov/dods

FNMOC / GODAE: usgodae.org

NCEP: nomad2.ncep.noaa.gov

NASA / GSWP: voda.gsfc.nasa.gov:9090

NASA / NSIPP: beta.gsfc.nasa.gov:9090

plus activity at centers in France, Britain (BADC), and Japan...


What to expect

Continuing refinements to existing code

- uncovering bugs (some suspected to still exist)

- improving documentation where needed

- addressing usability issues

Station data support

- Finishing up BUFR I/O interface in GrADS

- Adding daily station reports to COLA public server

- Releasing station-data-enabled GrADS client

- Working on interoperability with other archives and clients

Under consideration:

- built-in metric collection

- catalog server

- login-based authentication

- near-line data support

- listening for ideas


COLA thoughts on OPeNDAP"DODS" vs "OPeNDAP"

With apologies, waiting for 4.0 - don't want to confuse user community

New protocol

I see DAP 4.0 as a chance to reduce complexity and increase robustness

We all have limited development resources - keep size of protocol proportional

Interoperability means common ground, not sum of all features

Anagram / Server-side Java

Java is an ideal server platform - stable, cross-platform, easy to code

Consider Anagram framework + Java-DODS as servlet counterpart to DAP C++ CGI framework? Slot in custom functionality to a standard core - good for gateways, access control / resource limits, web services, etc.

Analysis protocol

Increasing usage at COLA - perceived by scientists as extremely valuable

Time to think about interoperability? Complex issues involved

Modeler's needs

Grid / ESG II

Authentication important for collaborative projects

Can OPeNDAP provide a way for small shops to get Grid-enabled?

grads-dods server an open-source tool for distributed data access and analysis joe wielgosz, brian...

Documents