Hydrologic Information System for Sharing SBRP Datasets

Ilya Zaslavsky

Spatial Information Systems LabSan Diego Supercomputer Center

UCSD

NIEHS SBRP Research Translation Workshop, LDEO, February 11-13, 2009

http://his.cuahsi.orghttp://hiscentral.cuahsi.orghttp://hydroseek.nethttp://river.sdsc.edu/ucsddashhttp://wron.net.au/DemosII/Modules/ODMKMLGatway.aspx

What is the CUAHSI HIS?CUAHSI: Consortium of Universities for the Advancement of Hydrologic Science, Inc. (120+ universities)CUAHSI HIS: NSF support through 2012, ~10 mil investedPartners:Academic: 11 NSF hydrologic observatories, CEO:P projects, LTER…Government: USGS, EPA, NCDC, NWS, state and localCommercial: Microsoft, ESRI, KistersInternational: Australia, UK, Malaysia, Singapore

An online distributed system to support the sharing of hydrologic data from multiple repositories and databases via standard water data service protocols;

software for data publication, discovery, access and integration.

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

Build a common window on water data using web services

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

http://his.cuahsi.org/odmdatabases.html

CUAHSI Observations Data Model

Information communication

• Water web pages • Water web services

HyperText Markup Language (HTML)

Water Markup Language (WaterML)

Water Data Services• Set of query functions • Returns data in WaterML

NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, USGS SNOTEL, ODM (multiple sites)

Test bed HISServers

Central HIS servers

ArcGIS

Matlab

IDL, R

MapWindow

Excel

Programming (Fortran, C, VB)

Desktop clients

Customizable web interface (DASH)

HTML - XMLW

SDL -SO

AP

Modeling (OpenMI)

Global search (Hydroseek)

WaterOneFlow Web Services, WaterML

Con

trolle

d vo

cabu

larie

s

Met

adat

aca

talo

gs

Ont

olog

y

ETL

serv

ices

HIS LiteServers

External data providers

Deployment to test beds

Other popular online clients

ODM DataLoader

Streaming Data Loading

Ontology tagging (Hydrotagger)

WSDL and ODM registration

Data publishing

ODMTools

Server config tools

HIS CentralRegistry & Harvester

Hydrologic Information System Service Oriented Architecture

Central HIS Data

Services Catalog

Semantic Tagging of Harvested Variables

Hydroseekhttp://www.hydroseek.net

Supports search by location and type of data across multiple observation networks including NWIS, Storet, and academic data

Against the NIH Syndrome2006:► CUAHSI HIS web services are discussed on the BASINS mailing list as a new way

to access hydrologic data. The list is mostly used by hydrologists and developers outside academia;

► NCDC develops ASOS web services following WaterML2007:► MOU with USGS; USGS is developing WaterML-compliant GetValues service;► GLEON uses an early version of ODM to develop their own schema (VEGA);► Phoenix LTER is developing ODM (in MySQL) and WaterML services (in Java);► A Google Earth-based client for CUAHSI web services is developed at CSIRO,

Australia;► Deployment to 11 hydrologic observatory test beds, + CBEO (CEOP project)2008:► KISTERS develops WaterML-compliant web services over their database, for a

client;► MapWindow open source GIS develops WaterOneFlow parsers;► Florida, Texas and Idaho use ODM and WaterOneFlow web services to provide

access to state data repositories; New Jersey is considering the same;► Another CEOP project, at UC-Davis, is implementing ODM (in Postgres) and web

services (in Java);► Stroud Water Research Center; WRON; AWI… many that we don’t know…► Now SBRP: data from UCSD, UA, more?

Summary• Generic method for managing and publishing observational

data– Supports many types of point observational data– Overcomes syntactic and semantic heterogeneity using a standard data

model and controlled vocabularies– Supports a national network of observatory test beds but can grow!

• WaterML is a standard language for consistently communicating water observations data from academic and government sources using web services

• National Water Metadata Catalog is the most comprehensive index of the nation’s water observations presently existing

Join the Water Data Federation!

HIS in SBRP: Why and How• WHY:

– Capacity building for academic and non-profit groups – Re-use and integration of collected data; discover other data collected

in your area; reduce duplication– Data support of partnerships– Emerging granting agency requirements

• E.g. NSF LTREB program (http://www.nsf.gov/pubs/2007/nsf07588/nsf07588.htm): " As part of the requirements for funding, projects must show how collected data will be shared broadly with the scientific community and the interested public.“

– Preserve contributions by different groups (beyond funding)– Publish data as part of publishing papers

• HOW:– HIS and SBRP communities have much in common– Understanding data models used in SBRP and how they relate to

the models we implement; extending controlled vocabularies and the search ontology; publishing data in a standard format; linking applications with the services

Consortium of Universities for the Advancement of Hydrologic Science, Inc.

An organization representing more than one hundred United States universities, receives support from the

National Science Foundation to develop infrastructure and services for the advancement of hydrologic

science and education in the U.S. http://www.cuahsi.org/

122 US Universities as of July 2008

Databases Analysis

Models

CUAHSI Hydrologic Information SystemGoal: Enhance hydrologic science by facilitating user access to more and better data for testing hypotheses and analyzing processes

• Advancement of water science is critically dependent on integration of water information– Querying nation’s repository of water data– Linking small integrated research sites (<100

km2) with global and continental models– Integrating data from multiple disciplines to

understand controls on hydrologic cycle• It is as important to represent hydrologic

environments precisely with data as it is to represent hydrologic processes with equationsRainfall

& SnowWater quantity

and quality Remote sensing Meteorology Soil water

SDSC Spatial Information Systems LabResearch and system development• Services-based spatial information

integration infrastructure• Mediation services for spatial data, query

processing, map assembly services• Long-term spatial data preservation• Spatial data standards and technologies for

online mapping (SVG, WMS/WFS)• Support of spatial data projects at SDSC

and beyond

services

In Geosciences (GEON, CUAHSI, CBEO,…)

In regional development (NIEHS SBRP, Katrina)

In Neurosciences (BIRN, CCDB)

http://spatial.sdsc.edu/lab/

Contact: zaslavsk@sdsc.edu