CUAHSI WaterML

Ilya Zaslavsky (SDSC),David Valentine (SDSC), Tim Whiteaker (UT-Austin)

/editors/

CUAHSI = Consortium of Universities for the Advancement of Hydrologic Sciences, Inc.;

Background

• CUAHSI HIS: NSF-supported collaborative project: UT Austin + SDSC + Drexel + Duke + Utah State (www.cuahsi.org/his/). PI: David R. Maidment (UT-Austin)

• A cyberinfrastructure project• Current focus: providing uniform access to

heterogeneous observations data, from different agencies• Plus an easy way to publish hydrologic observations data• And to assemble comprehensive observations databases for your

watershed, catchment, etc.

• From ad hoc SOAP wrappers for agency data to a consistent schema, agency buy-in, and support for multiple clients => WaterML

WaterOneFlow Web Services

Data access through web

services

Data storage through web

services

Dow

nlo

ads

Upl

oa

ds

Observatory servers

Workgroup HIS

SDSC HIS servers

3rd party servers

e.g. USGS, NCDC

GIS

Matlab

IDL

Splus, R

D2K, I2K

Programming (Fortran, C, VB)

Web services interface

Web portal Interface (HDAS)

Information input, display, query and output services

Preliminary data exploration and discovery. See what is available and perform exploratory analyses

HTML -XML WS

DL

- SO

AP

Hydrologic Information System Service Oriented Architecture

SupercomputerCenters:NCSA,TACC

Domain Sciences:

Unidata, NCARLTER, GEON

Government:USGS, EPA,

NCDC, USDA

Industry:ESRI, Kisters,

OpenMI

HISTeam

WATERSTestbed

WATERS Network Information System

CUAHSI HIS

The CUAHSI Community, HIS and WATERS

CUAHSI: 116 Universities (Nov. 2006)

HIS Team:Texas, SDSC,Utah, Drexel,

Duke

CUAHSI HIS as a mediator across multiple agency and PI data

• Keeps identifiers for sites, variables, etc. across observation networks

• Manages and publishes controlled vocabularies (in the Annex to the WaterML paper), and provides vocabulary/ontology management and update tools

• Provides common structural definitions for data interchange

• Provides a sample protocol implementation• Governance framework: a consortium of universities,

MOUs with federal agencies, collaboration with key commercial partners, led by renowned hydrologists, and NSF support for core development and test beds

WaterML design principles

• Driven largely by hydrologists; the goal is to capture semantics of hydrologic observations discovery and retrieval

• Relies to a large extent on the information model as in ODM (Observations Data Model), and terms are aligned as much as possible• Several community reviews since 2005

• Driven by data served by USGS NWIS, EPA STORET, multiple individual PI-collected observations

• Is no more than an exchange schema for CUAHSI web services

• The least barrier for adoption by hydrologists• A fairly simple and rigid schema tuned to the current

implementation• Conformance with OGC specs not in the initial scope

Point Observations Information Model

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• An observation series is an array of observations at a given site, for a given variable, with start time and end time• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value

Data Source

Network

Sites

ObservationSeries

Values

{Value, Time, Qualifier}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage, start, end (Daily or instantaneous)

206 cfs, 13 August 2006

Return network information, and variable information within the network

Return site information, including a series catalog of variables measured at a site with their periods of record

Return time series of values

Observations Data Model Schema (version 4.0)

Data Source and Network Sites Variables Values Metadata

Depth of snow pack

Streamflow

Landuse, Vegetation

Windspeed, Precipitation

Controlled Vocabulary Tables

e.g. mg/kg, cfs

e.g. depth

e.g. Non-detect,Estimated,

A site is a point location where one or more variables are measured

A data source operates an observation network A network is a set of observation sites

Metadata provide information about the context of the observation.A variable is a property describing the flow or quality of water

A value is an observation of a variable at a particular time

From Ernest To, David Maidment, CRWR

Challenges… (1/2)

• Sites• STORET has stations, and measurement points, at various offsets…• Site metadata lacking and inconsistent (e.g. 2/3 no HUC info, 1/3 no state/county info);

agency site files need to be upgraded to ODM…• A groundwater site is different than a stream gauge…

• Censored values• Values have qualifiers, such as “less than”, “censored”, etc. – per value. Sometimes

mixed data types.. • Units

• There are multiple renditions of the same units, even within one repository• There may be several units for the same parameter code (STORET)• If no value recorded – there are no units??• Unit multipliers

• E.g. NCDC ASOS keeps measurements as integers, and provides a multiplier for each variable

• Sources• STORET requires organization IDs (which collected data for STORET) in addition to site

IDs• Time stamps: ISO 8601

• A service to determine UTC offsets given lat/lon and date??

Challenges… (2/2)

• Values retrieval• USGS: by site, variable, time range• EPA: by organization-site, variable, medium, units, time range• NCDC: fewer variables, period of record applies to site, not to

seriesCatalog• Variable semantics

• Variable names and measurement methods don’t match• E.g. NWIS parameter # 625 is labeled ‘ammonia + organic nitrogen‘,

Kjeldahl method is used for determination but not mentioned in parameter description. In STORET this parameter is referred to as Kjeldahl Nitrogen.

• One-to-one mapping not always possible• E.g. NWIS: ‘bed sediment’ and ‘suspended sediment’ medium types vs.

STORET’s ‘sediment’.

Ontology tagging, semantic mediationContolled vocabularies are in Annex A

- From different database structures, data collection procedures, quality control, access mechanisms to uniform signatures … Water Markup Language- Tested in different environments- Standards-based- Can support advanced interfaces via harvested catalogs- Accessible to community- Templates for development of new services- Optimized, error handling, memory management, versioning, run from fast servers- Working with agencies on setting up services and updating site files

NWIS Daily Values (discharge), NWIS Ground Water, NWIS Unit Values (real time), NWIS Instantaneous Irregular Data, EPA STORET, NCDC ASOS, DAYMET, MODIS, NAM12K, ODM

WaterML key elements

• Response Types

– SiteInfo

– Variables

– TimeSeries

• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries

• values

– queryInfo

GetValues

GetVariableInfo

GetSiteInfo

variables

variablesResponse

variable

1

many

timeSeriesqueryInfo

criteria

timeSeriesResponse

variable

sourceInfo

queryURL

values

site queryInfo

criteria

sitesResponse

seriesCatalog siteInfo

queryURL

variable

series

variableTimeInterval

1

many

Structure of responses

DatasetInfoTypeSiteInfoType

Elements Defining Spatial Location

for observation sites for continuous surfaces

(other site information)

SourceInfoType

GeogLocationType

(other dataset information)

LatLonPointType

childelements

GeogLocationType

LatLonBoxType

LatLonPointType

SiteInfoResponseType

• Namespaces

• queryInfo

• site

Network

Sites

Variables

queryInfo example

userparameters

query URL

• Parameters sent to service

• URLs called (if external resource)

siteInfo

• Name

• Site Code

• Location

geoLocation

• geogLocation – geographic coordinates– LatLon point– LatLon box

• localSiteXY – projected coordinates

series

• variable – what is measured• valueCount – how many measurements• variableTimeInterval – when is it measured

TimePeriodType

variable

• variableCode – global identifier

• variableName

• units

Sites

Variables

Values

TimePeriodType

variableTimeInterval• TimePeriodType – date range (including

“last n days”

• TimeInstantType – single measurement

Compare with…

SiteInfo responsequeryInfo

site

name

code

location

seriesCatalog

variables

what

how many

when

TimePeriodType

VariablesResponseType

• variable – same as in series element

• Code, name, units Sites

Variables

Values

TimeSeriesResponseType

• queryInfo

• timeSeries– sourceInfo – “where”– variable – “what”– values

Sites

Variables

Values

sourceInfo• SiteInfoType

– Same as siteInfo element– code, name, location

• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType

values

• Each time series value recorded in value element

• Timestamp, plus metadata for the value, recorded in element’s attributes

ISO Time

valuequalifier

Compare with…

value metadata examples

• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset

– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode

TimeSeries responsequeryInfo

location

variable

values

Clients

• Tested with .Net and Java• Desktop clients:

Excel, Matlab, ArcGIS, VB.NET,more beingwritten

• Web client: DASH (Data Access System for Hydrology): http://river.sdsc.edu/DASH (beta)

DA

SH

AGS Server

IIS

Windows 2003 Server4 GB Ram1 TB Disk

Quad Core CPU

SQLServer

VS 2

005

WaterOneFlow Web Services

ArcGIS 9.2

GIS Data Mxd Service

OD

M L

oad

er

OD

M t

ools

OD

M

Current Deployment Current Deployment ArchitectureArchitecture

Direct DB connection

SQL Server

ODMs and catalogs. All instancesexposed as ODM (i.e. have standard ODM tables or views: Sites, Variables, SeriesCatalog, etc.)

NWIS-IID

NWIS-DV

ASOS

STORET

TCEQ

BearRiver

. . .

Spatial store

Geodatabase or collection of shapefilesor both

NWIS-IID points

NWIS-DV points

ASOS points

STORET points

TCEQ points

BearRiver points

. . .

My new ODM

My new points

More databases

More synced layers

DASH Web Application

Background layers

(can be in the same or separate spatial store)

WOF services

Web services from a common template

NWIS-IID WS

NWIS-DV WS

ASOS WS

STORET WS

TCEQ WS

BearRiver WS

. . .My new WS

More WS fromODM-WS template

USGS

NCDC

EPA

TCEQ

Web Configuration fileStores information about registered networks

MXDStores information about layers

WSDLs

, web

serv

ice U

RLs Connection

strings

Layer info,

symbology, etc.

ODMDataLoader

2

6

5

3

1

4

WORKGROUPHISSERVERORGANIZATION

STEPS FORREGISTERINGOBSERVATIO

NDATA

New network registration steps

Using the ODM DataLoader, load your data into a blank ODM instance (this will create all ODM tables, including Sites, Variables and SeriesCatalog that HIS application relies on)

Copy Web Services template to a new folder, edit the template web.config file to point to the new ODM, test to make sure the new service works as expected

Create a point layer (a feature class in GDB, or a shapefile) from the new ODM’s Sites table or from GetSites web service (using GetSitesTool – this will also test the service)

Add the point layer to the MXD document, specify symbology, scale-dependent rendering, etc.

Add information about the new ODM, the associated web service, and the associated point layer, to HIS configuration file (see the first slide for the exact content)

Restart the HIS service6

5

4

3

2

1

Near future

• Need further reviewed, based on initial implementation• Within and beyond OGC membership• Further engage with hydrologic observation groups at agencies• Internationalization (with CSIRO WRON, European WISE, others?)

• CUAHSI O&M profile?• Carry CUAHSI WaterML messages over O&M - need to talk with

Simon about it… • Test it within an initiative (e.g. Oceans IE, or Water Data

Interoperability Testbed, OWS-5? )• Have a list of suggestions for versions after 1.0 (including linear

referencing, GetCapabilities, WFS integration, handling multiple siteCodes and variableCodes in a single call) – more suggestions are welcome

• Divorcing from implementation? Looking for golden middle in standard complexity…

OGC Harmonization Best Practices

• WaterML text includes steps for harmonizing with GML/O&M

• Align spatial feature descriptions (e.g. using gml:Point, gml:Envelope)

• Align service signatures (getCapabilities)• Align terminology with O&M

• Provides guidance to other communities on harmonization steps

Motion

• EO/NRE WG recommends to the TC that the CUAHSI WaterML (document 07-041) be released as an OGC Discussion paper• Pending minor editorial changes• Proposed• Second: Ben Domenico• Unanimous

• Roadmap:• Reporting initial implementation experience• CUAHSI WaterML 1.1, aligning with O&M• Testing within an OGC initiative