Dvoy Networking Ideas

OpenGIS Web Services

• Mission: Definition and specification of geospatial web services.• A Web service is an application that can be published, located, and dynamically

invoked across the Web.• Applications and other Web services can discover and invoke the service.• The sponsors of the Web services initiative include

– Federal Geographic Data Committee– Natural Resources Canada– Lockheed Martin– National Aeronautics and Space Administration– U.S. Army Corps of Engineers Engineer Research and Development Center– U.S. Environmental Protection Agency EMPACT Program– U.S. Geological Survey– US National Imagery and Mapping Agency.

• Phase I - February 2002 – Common Architecture: OGC Services Model, OGC Registry Services, and Sensor

Model Language. – Web Mapping: Map Server- raster, Feature Server-vector, Coverage Server-image,

Coverage Portrayal Services. – Sensor Web: OpenGIS Sensor Collection Service for accessing data from a variety of

land, water, air and other sensors.

D-DADS Architecture

ARC/INFO

VirtualDataCube

ArcSDETranslator

OLAPService

Provider

DataCube

LegacyDatabase

CustomOLAP

Translator

DataCube

SQLDatabase

OLAP ServiceProvider

GISTable

OLAP

StandardizedDescription &

Format

Database(SQL,

Oracle,etc.)

ArcSDETranslator

GISMap

DataProviders

Data Access andManipulation Tools

UserInteraction

DataCube

ArcIMS

The D-DADS Components

• Data Providers supply primary data to system, through SQL or other data

servers. • Standardized Description & Format populate and describe the data

cubes and other data types using a standard metadata describing data

• Data Access and Manipulation tools for providing a unified interface to data cubes, GIS data layers, etc. for accessing and processing (filtering, aggregating, fusing) data and integrating data into virtual data cubes

• Users are the analysts who access the D-DADS and produce knowledge from

the data

The multidimensional data access and manipulation component of D-DADS will be implemented using OLAP.

Interoperability

“the ability to freely exchange all kinds of spatial information about the Earth and about objects and phenomena on, above, and below the Earth’s surface; and to cooperatively, over networks, run software capable of manipulating such information.” (Buehler & McKee, 1996)

Such a system has two key elements:

• Exchange of meaningful information

• Cooperative and distributed data management

One requirement for an effective distributed environmental data system is interoperability, defined as,

On-line Analytical Processing: OLAP

• A multidimensional data model making it easy to select, navigate, integrate and explore the data.

• An analytical query language providing power to filter, aggregate and merge data as well as explore complex data relationships.

• Ability to create calculated variables from expressions based on other variables in the database.

• Pre-calculation of frequently queried aggregated values, i.e. monthly averages, enables fast response time to ad hoc queries.

User Interaction with D-DADS

Query

Data View(Table, Map, etc.)

Distributed Database

XML data

XML data

Metadata Standardization

• The Supersite Data Management Workgroup

• NARSTO

• FGDC

Metadata standards for describing air quality data are currently being actively pursued by several organizations, including:

Potential D-DADS Nodes

The following organizations are potential nodes in a distributed data analysis and dissemination system:

• CAPITA

• NPS-CIRA

• EPA Supersites- California- Texas- St. Louis

Summary

In the past, data analysis has been hampered by data flow resistances. However, the tools and framework to overcome each of these resistances now exist, including:

• World Wide Web• XML• OLAP• OpenGIS• Metadata standards

Incorporating these tools will initiate a distributed data analysis and dissemination system.

‘Global’ and ‘Local’ AQ Analysis

• AQ data analysis needs to be performed at both global and local levels

• The ‘global’ refers to regional national, and global analysis. It establishes the larger-scale context.

• ‘Local’ analysis focuses on the specific and detailed local features

• Both global and local analyses are needed for for full understanding.

• Global-local interaction (information flow) needs to be established for effective management.

National and Local AQ Analysis

Data Re-Use and Synergy

• Data producers maintain their own workspace and resources (data, reports, comments).

• Part of the resources are shared by creating a common virtual resources.

• Web-based integration of the resources can be across several dimensions:Spatial scale: Local – global data sharing

Data content: Combination of data generated internally and externally

• The main benefits of sharing are data re-use, data complementing and synergy.

• The goal of the system is to have the benefits of sharing outweigh the costs.

Content

Content

User

User

User

LocalLocal

GlobalGlobal

Virtual Shared Resources

Data, KnowledgeTools, Methods

User

User

Shared part of resources

Integration for Global-Local Activities

Global Activity Local Benefit

Global data, tools Improved local productivity

Global data analysis Spatial context; initial analysis

Analysis guidance Standardized analysis, reporting

Local Activity Global Benefit

Local data, tools Improved global productivity

Local data analysis Elucidate, expand initial analysis

Identify relevant issues Responsive, relevant global analysis

Global and local activities are both needed – e.g. ‘think global, act local’

‘Global’ and ‘Local’ here refers to relative, not absolute spatial scale

Content Integration for Multiple Uses (Reports)

Data from multiple measurements are shared by their providers or custodiansData are integrated, filtered, aggregated and fused in the process of analysisReports use the analysis for Status and Trends; Exposure Assessment; Compliance …

The creation of the needed reports requires data sharing and integration from multiple sources.

Federated Data Warehouse Features

• As much as possible, data should reside in their respective home environment. ‘Uprooted’ data in decoupled databases tend to decay i.e. can not be easily updated, maintained, enriched.

• Data Providers would need to ‘open up’ their SQL data servers for limited data subsets and queries, in accordance with a ‘contract’. However, the data structures of the Providers will not need to be changed.

• Data from the providers will be transferred to the ‘federated data warehouse’ through (1) on-line DataAdapters, (2) Manual web submission and (3) Semi-automated transfer from the NARSTO archive.

• Retrieval of uniform data from the data warehouse facilitates integration and comparison along the key dimensions (space, time, parameter, method)

• The open architecture data warehouse (see Web Services) promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..

Navigation Service

DVoy: Components and Data Flow

Legacy Data

Publish

(DataSet)

DataSet Records

• Provider Descript.

• Service Descript.

• Measure Access

Find (Measure)

DataSet Recs

Selected Measure Record

• Dvoy

Data Wrapping

WebServiceBind (Measure, FocusCube)

Data Delivery

WebServiceFocusCube, GlobCursor,

Measure, Granule

Time Chart

Layered Map

DataToView

DataForCursorAndView

Catalog Service Data Services

Data Delivery

WebService

Presentation Services

Data Focus Range

Rendering

Cursor

Viewer Layers

Dim1: LonDim2: Lat

Data provided by each dimension of a View:

Dim1.Type, Dim1.Min, Dim1.Max

Dim2.Type, Dim2.Min, Dim2.Max

….

Current Dim.Types:

Latitude, Longitude, DateTime, Elevation

Federated Data Services Architecture

XML WebServices

Satellite

Vector

GIS Data

XDim Data

OLAPCube

SQLTable

HTTPServices

Text Data

WebPage

TextData

Scatter Chart

Text, Table

Data View & Process TierLayered Map

Cursor

Data Warehouse Tier

Data View

Manager

Connection

Manager

Data Access

Manager

Cursor-Query

Manager

OpenGISServices

Data are rendered by linked Data Views (map, time, text)

Distributed data of multiple types (spatial, temporal text)

The Broker handles the views, connections, data access, cursor

Time Chart

Dvoy Federated Information System

• Dvoy offers a homogeneous, read-only access mechanism to a dynamically changing collection of heterogeneous, autonomous and distributed information sources.

• Data access uses a global multidimensional schema consisting of spatial, temporal and parameter dimensions

• The uniform global schema is suitable for data browsing and online analytical processing, OLAP

• The limited global query capabilities yield slices along the spatial, temporal and parameter dimensions of the multidimensional data cubes.

Architecture of DATAFED Federated Data SystemAfter Busse et. al., 1999

• The main software components of Dvoy are wrappers, which encapsulate sources and remove technical heterogeneity, and mediators, which resolve the logical heterogeneity.

• Wrapper classes are available for geo-spatial (incl. satellite) images, SQL servers, text files,etc. The mediator classes are implemented as web services for uniform data access to n-dimensional data.

Integration Architecture (Ullman, 1997)

• Heterogeneous sources are wrapped by software that translates between the sources local language, model and concepts and the shared global concepts

• Mediators obtain information from one or more components (wrappers or other mediators) and pass it on to other mediators or to external users.

• In a sense, a mediator is a view of the data found in one or more sources; it does not hold the data but it acts as it it did. The job of the mediator is to go to the sources and provide an answer to the query.

Federated PM and Haze Data Warehouse Project

a sub- project of

(enter your sticker & logo here )

Nov 20, 2001, RBH

St. Louis Midwest Supersite Project

Regional Planning OrganizationRPO

EPA Supersites SupSite

NARSTO PMNARSTO

EPA Division1, Division2, Division2 EPA

Me and my dog for our aerosol projectMe

PM/Haze Data Flow in Support of AQ Management

• There are numerous organizations in need of data relevant to PM/Haze

• Most interested parties (stakeholders) are both producers and consumers of PM and haze data

• There is a general willingness to share data but the resistances to data flow and processing are too high

RPO

RPO

RPO

Regional Planning Orgs

FLM

FLM

FLM

Federal Land Managers

EPA

EPA

EPA

EPA Regul. & Research

Industry

AcademicNARSTO

Other: Private, Academic

SuperSite

Shared PM/Haze

Data

• PM and haze data are used for may parts of AQ management, mostly in form of Reports

• The variety of pertinent (ambient, emission) data come from many different sources

• To produce relevant reports, the data need to be ‘processed’ (integrated, filtered aggregated)

Scientific and Administrative Rationale for Resource Sharing

Scientific Rationale:• Regional haze and its precursors have a 1000-10000 km airshed.• (Smoke, Dust, Haze) – Data integration • Substantial fraction of haze originates from natural sources or

from out-of-jurisdiction man-made sources• Cross-RPO data and knowledge sharing yields better operational

and science support to AQ management

Management Rationale:

• Haze control within some RPOs cannot yield • Data sharing saves money and ….

A Strategy for the Federated PM/Haze Data Warehouse

• Negotiate with the data providers ‘open up’ their data servers for limited, controlled, access in accordance with clear ‘access contract’ with the Federated Warehouse

• Design an interface to the warehoused datasets that has simple data access and satisfies the data needs of most integrating users.(oxymoron ????)

• Facilitate the the development of shared value-adding processes (analysis tools, methods) that refine the raw data to useful knowledge

Three-Tier Federated Data Warehouse Architecture

(Note: In this context, ‘Federated’ differs from ‘Federal’ in the direction of the driving force. Federated meant to indicate a driving force for sharing from ‘bottom up’ i.e. from the members, not dictated from ‘above’, by the Feds)

1. Provider Tier: Back-end servers containing heterogeneous data, maintained by the federation members

2. Proxy Tier: Retrieves designated Provider data and homogenizes it into common, uniform Datasets

3. User Tier: Accesses the Proxy Server and uses the uniform data for presentation, integration or processing

Provider Tier

Heterogeneous data in distributed SQL Servers

Proxy Tier

Data homogenization, transformation

Federated Data Warehouse

User Tier

Data presentation, processing

Federated Data Warehouse Interactions

• The Provider servers interact only with the Proxy Server in accordance with the Federation Contract– The contract sets the rules of interaction (accessible data subsets, types of queries)– Strong server security measures enforced, e.g. through Secure Socket layer

• The data User interacts only with the generic Proxy Server using flexible Web Services interface– Generic data queries, applicable to all data in the Warehouse (e.g. data sub-cube by space, time, parameter)– The data query is addressed to the Web Service provided by the Proxy Server– Uniform, self-describing data packages are passed to the user for presentation or further processing

SQLDataAdapter1

CustomDataAdapter

SQLDataAdapter2

SQLServer1

SQLServer2

LegacyServer

Presentation

Data Access & Use

Provider Tier Heterogeneous Data

Proxy Tier

Data Homogenization, etc.

Member ServersProxy Server

User Tier

Data Consumption

Processing

Integration

Federated Data Warehouse

Fire Wall, Federation ContractWeb Service, Uniform Query & Data

‘Global’ and ‘Local’ AQ Analysis

• AQ data analysis needs to be performed at both global and local levels

• The ‘global’ refers to regional national, and global analysis. It establishes the larger-scale context.

• ‘Local’ analysis focuses on the specific and detailed local features

• Both global and local analyses are needed for for full understanding.

• Global-local interaction (information flow) needs to be established for effective management.

National and Local AQ Analysis

Data Model Ray Plante, Virtual Obs

• What’s the difference between Data Models and Metadata? Intertwined– metadatum: a datum with a name or semantic tag that refers to the data– data model: a description of the relationships between metadata

• structural & logical relationships between compound objects & their components• operations that can be performed on them (really -

– framework: the architecture/process used to define metadata/data models that enables their ready use in applications

• Formalized data modeling process– encourages community involvement for defining standard models & metadata– structure enables easy verification, dissemination, & automated use– “standard” metadata should point directly to components of the “standard”

models– allow groups to define metadata independent of a “standard” metadata

• Practical Difference?– data model captures as complete a picture of a concept as possible– metadata represents the instantiation of portion of the model’s components

• Data access through a data model (Wrapper Classes for each data model)

Integration for Global-Local Activities

Global Activity Local Benefit

Global data, tools => Improved local productivity

Global data analysis => Spatial context; initial analysis

Analysis guidance => Standardized analysis, reporting

Local Activity Global Benefit

Local data, tools => Improved global productivity

Local data analysis => Elucidate, expand initial analysis

Identify relevant issues => Responsive, relevant global work

Global and local activities are both needed – e.g. ‘think global, act local’

‘Global’ and ‘Local’ here refers to relative, not absolute scale

Federated Data System Features

• Data reside in their respective home environment where it can mature. ‘Uprooted’ data in centralized databases are not easily updated, maintained, enriched.

• Abstract (universal) query/retrieval facilitates integration and comparison along the key dimensions (space, time, parameter, method)

• The open data query based on Web Services promotes the building of further value chains: Data Viewers, Data Integration Programs, Automatic Report Generators etc..

• The data access through the Proxy server protects the data providers and the data users from security breaches, excessive detail

Integration for Global-Local Activities

Global Activity Local Benefit

Global data & analysis Spatial context; initial analysis

Analysis guidance Standardized analysis, reporting

Local Activity Global Benefit

Local data & analysis Elucidate, expand initial analysis

Identify relevant issues Responsive, relevant global analysis

Global and local activities are both needed – e.g. ‘think global, act local’

‘Global’ and ‘Local’ here refers to relative, not absolute spatial scale

Data Re-Use and Synergy

• Data producers maintain their own workspace and resources (data, reports, comments).

• Part of the resources are shared by creating a common virtual resources.

• Web-based integration of the resources can be across several dimensions:Spatial scale: Local – global data sharing

Data content: Combination of data generated internally and externally

• The main benefits of sharing are data re-use, data complementing and synergy.

• The goal of the system is to have the benefits of sharing outweigh the costs.

Content

Content

User

User

User

LocalLocal

GlobalGlobal

Virtual Shared Resources

Data, KnowledgeTools, Methods

User

User

Shared part of resources

Federated Information System

• Providers maintain their own workspace and resources (data, tools, reports)

• Part of the private resources are exposed as shared (federated) resources

• The Federation facilitates finding, accessing and usage of the shared resources

Data sharing federations: • Open GIS Consortium (GIS data layers)• NASA SEEDS network (Satellite data)• NSF Digital Government • EPA’s National Env. Info Exch. Network.

InfoData

Shared (Federated) Resources

Data, Services, Tools, Methods

SharedPrivate

Other Federations

Providers/Users

Data Federation Concept and the FASNET Network

Schematic representation of data sharing in a federated information system.Based on the premise that providers expose part of their data (green) to others

Schematics of the value-adding network proposed for FASTNETComponents embedded in the federated value network

Data Acquisition and Usage Value Chain

Monitor StoreData 1

Monitor StoreData 2

Monitor StoreData n

Monitor StoreData m

IntData1

IntDatan

IntData2 Virtual Int. Data

Processes of the Information Value Chain

•

Informing Knowledge

ActionProductive Knowledge

Information

Organizing

Grouping Classifying Formatting Displaying

Analyzing

SeparatingEvaluating

Interpreting

Synthesizing

Judging

Options Quality Advantages

Disadvantages

Deciding

Matching goals, Compromising

Bargaining Deciding

CIRA VIEWS Langley IDEA AQ ManagerWG Summary Rpt

Data

(after Taylor, 1975)

Examples:

Data Flow and Processing

Next Process

Next Process

Why? How?

When? Where?CATT: A Community Tool!

Part of an Analysis Value Chain

Aerosol Data

Collection IMP. EPA

Aerosol Sensors

Integration VIEWS

Integrated AerData

AEROSOL

Weather Data

Assimilate NWS

Gridded Meteor.

Trajectory ARL

Traject.Data

TRANSPORT

TrajData Cube

Aggreg. Traject.

AerData Cube

CATT

Aggreg.Aerosol

CATT-In CAPITA

CATT-In CAPITA

There!

Not There! Further Analysis

GIS

Grid Processing

Emission

Comparison

Fast Aerosol Sensing Tools for Natural Event Tracking FASTNET

Analysts Console

Community Website

Distributed Programming: Interpreted and Compiled

• Web services allow processing of distributed data– Data are distributed and maintained by their custodians,

– Processing nodes (web-services) are also distributed

– ‘Interpreted’ web-programs for data processing can be created ad hoc by end users

• However, ‘interpreted’ web programs are slow, fragile and uncertain– Slow due to large data transfers between nodes

– Fragile due to instability of connections

– Uncertain due to failures of data provider and processing nodes

• One solution is to ‘compile’ the data and processing services– Data compilation transforms the data for fast, effective access (e.g. OLAP)

– Web service compilation combines processes for effective execution

• Interpreted or compiled?– Interpreted web programs are simpler and up to date but slow, fragile, uncertain

– Compiled versions are more elaborate and latent but also faster and more robust

– Frequently used datasets and processing chains should be compiled and kept current

Interpreted and Compiled Service

Point Access

Point Grid

Grid Render

Point Render

PtGrid Overla

y

Point Access

Point Grid

Grid Render

Point Access

Point Render

PtGrid Overlay

Interpreted Service

• Processes distributed

• Data flow on Internet

Compiled Service

• Processes in the same place

• Data flow within aggregate service

• Controllers, e.g. zoom can be shared

Data Flow

Control Flow

Voyager: The Program

• The Voyager program consists of a stable core and adoptive input/output section• The core executes the data selection, access portrayal tasks• The adoptive, abstract I/O layer connects the core to evolving web data, flexible

displays and to the a configurable user interface:– Wrappers encapsulate the heterogeneous external data sources and homogenize the access– Device Drivers translate generic, abstract graphic objects to specific devices and formats – Ports expose the internal parameters of Voyager to external controls

Data Sources

Controls

Displays

Voyager Core

Data Selection

Data Access

Data Portrayal

Adoptive Abstract I/O Layer

Dev

ice

Dri

vers

Ports

Wra

pp

ers

Dvoy_Services: Generic Software components

Webservice

Param 1

Param2

Service state

Webservice Adaptor

User Interface Module

Controller

state I/o ports

state I/o ports

Web service calls

Web serviceOutput data

Web serviceInput data

User Interface Module

UIM extracts relevant UI parameters from STATE

User changes UI parameters

UIM transmits modified UI parameters to STATE

Service Chain STATE Module

Contains the state params for all services in the chain

Has ports for getting/setting state params

Service Adopter Module

Gets input data from upsteam service

Gets service params from STATE

Make service call

Service Adopter ModuleGets input data from upsteam service

Gets service params from STATE

Make service call

Web service Service Module

Gets service call from Adopter module

Executes service

Returns output data

PointAccess->Grid->GridRender Service Chain

• The service chain interpreter make ONLY 2 sequential calls, stated in the data flow program:– GetMapPointDataAdaptor– RenderMapviewPoint Adaptor

GetMapPointDatadataset_abbr: IMPROVE

Param_abber SOILf

datatime: 2001-04-16

sql_filter:

RenderMapviewPoint

dataset_url:

output_format:

out_image_width:

Etc…..

Service state

GetMapPointData Adaptor

RenderMapviewPoint Adaptor

GetMapPointData Selector

RenderMapviewPoint Selector

state I/o ports

state I/o ports

Web service calls

Web serviceOutput data

Service state

PointAccess->Grid->GridRender Service Chain

• The service chain interpreter make ONLY 3 sequential calls, stated in the data flow program:– GetMapPointDataAdaptor– GridMapviewPointAdaptor– RenderMapviewGridAdaptor

GetMapPointDatadataset_abbr: IMPROVE

Param_abber SOILf

datatime: 2001-04-16

sql_filter:

RenderMapviewGriddataset_url:

output_format:

out_image_width:

Etc…..

GetMapPointData Adaptor

RenderMapviewGrid Adaptor

GetMapPointData Selector

RenderMapviewGrid Selector

GridMapviewPoint Selector

GridMapviewPointdataset_url:

output_format:

out_image_width:

Etc…..

GridMapviewPoint Adaptor

state I/o ports

state I/o ports

Web service calls

Web serviceOutput data

Service state Service state Service state

VOYAGER Web Services

Layered Map

Time ChartVector

GIS Data

XDim DataSQL Tables

WebImages

Publish, Find, Bind

Catalog, Data & Tools

Uniform Access

Scatter Chart

S u p p o r tCoordination T e c h n o l o g i e s

Users

Select, Overlay, Explore; Multidimensional data

Providers

Maintain distributed data; Heterogeneous coding , access

Voyager Web Services

Homogenize data access Catalog, access, transform data

C O M M U N I T Y

Services Program Execution:

Reverse Polish Notation

Writing the WS program:- Write the program on the command line of a URL call- Services are written sequentially using RPN - Replacements

Connector/Adaptor: - Reads the service name from the command line and loads its WSDL- Scans the input WSDL - The schema walker populates the service input fields from:

- the data on the command line - the data output of the upstream process - the catalog for the missing data

Service ExecutionFor each service

Reads the command line, one service at a timePasses the service parameters to the above Connector/Adopter, which prepares

the serviceExecutes the service

It also handles the data stack for RPN

Lecture Notes by M. SmallPg 18-19; 23-24

Pg. 25-26; 23-24N2

Value Network_Anderson