CUAHSI WaterOneFlowWeb Services

By Tim WhiteakerCE 394K.2 Hydrology

1 February 2007

Outline

• What are Web Services• What is WaterOneFlow• What is WaterML• How do we use HIS Analyst to access

WaterOneFlow

Programs Use Functions• Functions are pieces of code that perform

a specific task

Program

Mathlibrary

AddMultiplySquareRoot

Functions:

What’s the square root

of 16?

16

4

result = Math.SquareRoot(16)

Web Services Let Computers Share Functions

Program

Mathlibrary

AddMultiplySquareRoot

Functions:What’s the square root of 16

The answer is 4

• Local programs take advantage of remote resources

• Works just like local functionresult = Service.SquareRoot(16)

ServiceClient

How Do Web Services Work

• SOAP (Simple Object Access Protocol) is a protocol for exchanging messages over a network

• WSDL (Web Services Description Language) is a language for describing what a web service can do

How Do Web Services Work

Client

Server

SOAP WSDL(This is what I

can do)

SOAP is like speaking the same languageWSDL is like a contract

CUAHSI Web Services (WaterOneFLow)

• Standard mechanism for flow of hydrologic data between hydrologic data servers (databases) and users.

• Provides data access to– USGS NWIS– EPA STORET– NCEP North American Model (NAM)– NASA MODIS– Daymet– NWS ASOS

http://water.sdsc.edu/waterOneFlow/

Water Data Web Sites

NWISWeb site output# agency_cd Agency Code# site_no USGS station number# dv_dt date of daily mean streamflow# dv_va daily mean streamflow value, in cubic-feet per-second# dv_cd daily mean streamflow value qualification code## Sites in this file include:# USGS 02087500 NEUSE RIVER NEAR CLAYTON, NC#agency_cd site_no dv_dt dv_va dv_cdUSGS 02087500 2003-09-01 1190USGS 02087500 2003-09-02 649USGS 02087500 2003-09-03 525USGS 02087500 2003-09-04 486USGS 02087500 2003-09-05 733USGS 02087500 2003-09-06 585USGS 02087500 2003-09-07 485USGS 02087500 2003-09-08 463USGS 02087500 2003-09-09 673USGS 02087500 2003-09-10 517USGS 02087500 2003-09-11 454

Time series of streamflow at a gaging station

Observation Stations

Ameriflux Towers (NASA & DOE) NOAA Automated Surface Observing System

USGS National Water Information System NOAA Climate Reference Network

Map for the US

WaterOneFlow• Consistent Query Format • Consistent Data Format

WaterOneFlow Methods

• (Almost) the same for all networks• Methods

– GetSiteInfo – Give me info about a site– GetVariableInfo – Give me info about a

variable– GetValues – Give me a time series for a

variable at a site

GetSiteInfo

• Input– Network name and

site code– Authorization token

• Output– Site name– Site location– Variables measured at

the site• Variable code• Period of record

response = service.GetSiteInfo("NWIS:08158000", "")

where

GetVariableInfo

• Input– Network name and

variable code– Authorization token

• Output– Variable name– Variable description– Variable units

response = service.GetVariableInfo("NWIS:00060", "")

what

GetValues

• Input– Network name and

location (e.g., site code)

– Network name and variable code

– Start datetime– End datetime– Authorization token

• Output– Site info– Variable info– Time series

response = service.GetValues("NWIS:08158000", "NWIS:00060", _ "2006-12-31T01:00:00", "2006-12-31T08:00:00", "")

where what

when

The “Where” in GetValues

• Works for sites, or geographic locations• NetworkName:SiteCode

– NWIS:08158000

• GEOM:POINT(Longitude Latitude)– GEOM:POINT(-113 35)

• GEOM:BOX(WLon SLat,ELon NLat)– GEOM:BOX(-108 45,-107 46)

Locations

Variable Codes

Date Ranges

Extract – Transform – Load

GetSiteInfoGetVariableInfoGetValues

WaterOneFlowWeb Service

Client

STORET

NAMNWIS

DataRepositories

Data

DataData

EXTRACTTRANSFORMLOAD

WaterML

WaterML – The Output of WaterOneFlow

• Standardized for all data sources• Formats

– Object– XML (for programs that can’t handle objects)

• Defined in XML

XML – A Primer

• The Extensible Markup Language (XML) is a W3C-recommended general-purpose markup language that supports a wide variety of applications. – Wikipedia

• XML represents data• XML is both human and machine readable

<site> <name>Mansfield Dam</name></site>

XML Structure

<site></site>

<site></site>

<site />

OR

OR

element

start tag end tag

• Example – Let’s describe a streamflow site

element name = “site”

Elements can have Children

<site> <name>Mansfield Dam</name></site>

element

child element

element text

Elements can have Attributes

<site> <name>Mansfield Dam</name> <siteCode network=“NWIS”>08154510</siteCode></site>

attribute name attribute value

XML Nesting

<site> <name>Mansfield Dam</name> <siteCode network=“NWIS”>08154510</siteCode> <location> <latitude>30.39</latitude> <longitude>97.91</longitude> </location></site>

WaterML

• XML for describing water data• Defined at

• WaterOneFlow returns data in WaterML format

http://water.sdsc.edu/waterOneFlow/documentation/schema/

Point Observations Information ModelData Source

Network

{Value, Time, Qualifier}

USGS

Streamflow gages

Neuse River near Clayton, NC

Discharge, stage (Daily or instantaneous)

206 cfs, 13 August 2006

• A data source operates an observation network• A network is a set of observation sites• A site is a point location where one or more variables are measured• A variable is a property describing the flow or quality of water• A value is an observation of a variable at a particular time• A qualifier is a symbol that provides additional information about the value

Sites

Variables

Values

http://www.cuahsi.org/his/webservices.html

WaterML Response

SiteInfoResponseType

VariablesResponseType

TimeSeriesResponseType

WaterML Key Elements

• Response Types

– SiteInfo

– Variables

– TimeSeries

• Key Elements– site– sourceInfo– seriesCatalog– variable– timeSeries

• values

– queryInfo

GetValues

GetVariableInfo

GetSiteInfo

SiteInfoResponseType

• Namespaces• queryInfo• site

Network

SitesVariables

queryInfo

• Parameters sent to service• URLs called (if external resource)

queryInfo – Another Example

userparameters

query URL

site

• siteInfo – info about the site• seriesCatalog – catalogs of variables

measured

siteInfo

• Name• Site Code• Location

geoLocation

• geogLocation – geographic coordinates– LatLon point– LatLon box

• localSiteXY – projected coorindates

European Petroleum Survey Groupi.e., NAD 83, etc.

seriesCatalog

name web service location

original data sourceseries (variables)

series

• variable – what is measured• valueCount – how many measurements• variableTimeInterval – when is it measured

variable

• variableCode – global identifier• variableName• units

Sites

VariablesValues

variableTimeInterval

• TimeIntervalType – date range• TimeSingleType – single measurement• TimePeriodRealTimeType – last n days

ISO Time

• International Organization for Standardization• 1998-03-01T14:30:00 =

March 1, 1998, at 2:30 in the afternoon

SiteInfo Response, RecapqueryInfo

site

name

code

location

seriesCatalog

variables

what

how many

when

VariablesResponseType

• variable – same as in series element• Code, name, units Sites

VariablesValues

Additional variable Children

• variableDescription• valueType (enumeration)

– Field Observation– Sample– Model Simulation Result– Derived Value

• generalCategory – e.g., “water quality”• sampleMedium – e.g., “water column”

-- Many are optional

TimeSeriesResponseType

• queryInfo• timeSeries

– sourceInfo – “where”– variable – “what”– values

Sites

Variables

Values

sourceInfo• SiteInfoType

– Same as siteInfo element– code, name, location

• DataSetInfoType– For data continuous in space– LatLonPointType– LatLonBoxType

variable

• Same as previous variable element• code, name, units, etc.

values

• Each time series value recorded in value element

• Timestamp, plus metadata for the value, recorded in element’s attributes

ISO Time

valuequalifier

value Metadata Examples

• qualifiers• censorCode (lt, gt, nc)• qualityControlLevel (Raw, QC’d, etc.)• methodID• offset

– offsetValue– offsetUnitsAbbreviation– offsetDescription– offsetUnitsCode

TimeSeries Response, RecapqueryInfo

location

variable

values

Conclusions about WaterML

• Consistent Format• Includes lots of information• How do I use it?

HIS Analyst

HIS Analyst

• Desktop apps that use WaterOneFlow• Examples for various application

environments• HydroObjects – for apps that aren’t web

service enabled• Workbook available at:

http://www.cuahsi.org/his/docs/HIS-workbook-20061130.pdf

HIS Workbook

• Ingesting NWIS Data into Excel• Ingesting STORET Data into Excel• Ingesting Weather and Streamflow Data

into ArcGIS• Plotting MODIS Data with Matlab• Ingesting NWIS Data using VB.Net• Ingesting NWIS Data Using Java

Excel Demo

Weather Downloader

Historical Weather Data

Forecasted Weather Data

Streamflow

Other

Recap

• Computers share functionality through Web Services

HIS Analyst

Client WaterOneFlowWeb Service

GetSiteInfoGetVariableInfoGetValues

WaterMLTime SeriesRepository