The Problem: Integrating Data, Applications, and Client Devices

The key issue we try to solve is building the distributed computing infrastructure The key issue we try to solve is building the distributed computing infrastructure that can connectthat can connect• Legacy data archivesLegacy data archives• Executable codesExecutable codes• Real time data sourcesReal time data sources• Client tools for collaborationClient tools for collaboration

Audio/Video systems, whiteboard annotators, etc Audio/Video systems, whiteboard annotators, etc Various application-specific grids can be built out of the Various application-specific grids can be built out of the common infrastructurecommon infrastructure

• Science Grids Science Grids • Emergency planning, crisis response Emergency planning, crisis response

We choose certain fixed points for our We choose certain fixed points for our foundationsfoundations• Other standards where available: GIS standardsOther standards where available: GIS standards• Web Service standards: SOAP and WSDLWeb Service standards: SOAP and WSDL• Universal messaging substrate for SOAP and other messages: Universal messaging substrate for SOAP and other messages:

http://www.naradabrokeringhttp://www.naradabrokering

OGC Standards for GIS OpenGIS Consortium is an international group OpenGIS Consortium is an international group

that defines common formats for geospatial data that defines common formats for geospatial data and service standards.and service standards.

Main data format language is the XML-based Main data format language is the XML-based GeographyGeography Markup Language (GML)Markup Language (GML)..• GML is widely accepted as the universal encoding for GML is widely accepted as the universal encoding for

geo-referenced data. geo-referenced data. OGC defines services.OGC defines services.

• Some example OGC services includeSome example OGC services include Web Feature Service (WFS):Web Feature Service (WFS): for retrieving GML encoded for retrieving GML encoded

features, like faults, roads, county boundaries, GPS station features, like faults, roads, county boundaries, GPS station locations,….locations,….

Web Map Service (WMS):Web Map Service (WMS): for creating maps out of Web for creating maps out of Web FeaturesFeatures

http://www.opengeospatial.orghttp://www.opengeospatial.org

What Are Web Services? Web Services framework is a way

for doing distributed computing with XML.

• WSDL: Defines interfaces to functions of remote components. This is the programming interface.

• SOAP: Defines the message format that you exchange between components. This carries the messages over the network.

• Web Services are not web pages, CGI, servlets, applets.

XML provides cross-language support

Suitable for both human and application clients.

We are building data grid components in the same fashion.

• Allows us to build general purpose client environments like Web portals and command line tools.

http://www.w3.org/2002/ws

WebServer

DB

JDBC

Browser

WebServer

SOAP

Appl

SOAPWSDL

WSDL

WSD

LWSD

L

Service Oriented Architecture

Key SOA Features Key SOA Features (Following W3C’s WS Architecture working group.)(Following W3C’s WS Architecture working group.)

• SOAs are composed of services that present SOAs are composed of services that present programmatic access to resources to remote programmatic access to resources to remote client applications. client applications.

• Services communicate using messages. Services communicate using messages. Messages are usually encoded using SOAP.Messages are usually encoded using SOAP.

• SOAs are metadata rich. We must describe SOAs are metadata rich. We must describe service interfaces, provide descriptions of service interfaces, provide descriptions of services and publish them into registries. services and publish them into registries.

WFS

Information Service

- WSDL URL- NB Endpoint

- GetCapabilities - GetFeature- DescribeFeature

WMS

- GetCapabilities - GetMap- GetFeatureInfo `

Web Map Client

Publish

Find

Bind

Publish/Find/Bind paradigm in a GIS Grid

Service providers

Service requestors

Service registries

Service Oriented Architecture

NaradaBrokering (NB)NaradaBrokering (NB) NaradaBrokering is a distributed messaging NaradaBrokering is a distributed messaging

infrastructure.infrastructure.

It may be thought of a as topic-based It may be thought of a as topic-based publish/subscribe messaging system: publish/subscribe messaging system: • interested entities can register to a NaradaBrokering interested entities can register to a NaradaBrokering

node to send and receive messages on particular topics. node to send and receive messages on particular topics.

It supports multiple underlying transport It supports multiple underlying transport protocols, intelligent message routing, protocols, intelligent message routing, asynchronous messaging, reliable delivery, replay asynchronous messaging, reliable delivery, replay of events, compression/fragmentation, discovery of events, compression/fragmentation, discovery etc.etc.

http://www.naradabrokering.orghttp://www.naradabrokering.org

Pattern Informatics in a Grid Environment

PI in a Grid environment:• Hotspot forecasts are made using publicly available seismic records.

Southern California Earthquake Data Center Advanced National Seismic System (ANSS) catalogs

• Code location is unimportant, can be a service through remote execution• Results need to be stored, shared, modified• Grid/Web Services can provide these capabilities

Problems:• How do we provide programming interfaces (not just user interfaces) to

the above catalogs?• How do we connect remote data sources directly to the PI code.• How do we automate this for the entire planet?

Solutions:• Use GIS services to provide the input data, plot the output data• Use HPSearch tool to tie together and manage the distributed data

sources and code.

<?xml version="1.0" encoding="UTF-8"?><?xml version="1.0" encoding="UTF-8"?> <soap:Envelope xmlns:soap="http://www.w3..."><soap:Envelope xmlns:soap="http://www.w3..."> <soap:Header encodingStyle=“WSCTX URL"<soap:Header encodingStyle=“WSCTX URL"

mustUnderstand="true">mustUnderstand="true"> <<contextcontext xmlns=“ctxt schema“ timeout="100"> xmlns=“ctxt schema“ timeout="100"> <<context-idcontext-id>http..</context-id>>http..</context-id> <<context-servicecontext-service> http.. </context-service>> http.. </context-service> <<context-managercontext-manager> http.. </context-service>> http.. </context-service> <<activity-listactivity-list mustUnderstand="true" mustPropagate="true">mustUnderstand="true" mustPropagate="true"> <<p-servicep-service>http://../WMS</p-service>>http://../WMS</p-service> <<p-servicep-service>http://../HPSearch</p-service>>http://../HPSearch</p-service> </activity-list></activity-list> </context></context> </soap:Header></soap:Header>......

SOAP header

for Context

The Pattern Informatics GIS-SOA based workflow application

5,6: WMS starts a session, invokes HPSearch to run workflow script for PI Code with a session id

7,8,9: HPSearch runs the workflow script and generates output file in GML format (& PDF Format) as result

10: HPSearch writes the URI of the of the output file into Context

11: WMS polls the information from Context Service

12: WMS retrieves the generated output file by workflow script and generates a map

<context xsd:type="ContextType"timeout=“100"><context xsd:type="ContextType"timeout=“100"><context-service><context-service>http://.../HPSearch</ http://.../HPSearch</ context-context-

service>service>

<content> HPSearch associated additional data <content> HPSearch associated additional data generated during execution of workflow. </content>generated during execution of workflow. </content>

</context></context>

service associated

<context xsd:type="ContextType"timeout=“100"><context xsd:type="ContextType"timeout=“100"><context-service><context-service>http://.../WMS</ http://.../WMS</ context-service>context-service>

<activity-list mustUnderstand="true" <activity-list mustUnderstand="true" mustPropagate="true"> mustPropagate="true">

<service>http://.../WMS</service> <service>http://.../WMS</service>

<service>http://.../HPSearch</service> <service>http://.../HPSearch</service>

</activity-list> </activity-list>

</context></context>

session

<context xsd:type="ContextType"timeout=“100"><context xsd:type="ContextType"timeout=“100"><context-service><context-service>http://.../HPSearch</ http://.../HPSearch</ context-context-

service>service><parent-context>http://../abcdef:012345<parent-<parent-context>http://../abcdef:012345<parent-

context/>context/><content> <content> profile information related WMS profile information related WMS

</content></content>

</context></context>

user profile

<context xsd:type="ContextType"timeout=“100"><context xsd:type="ContextType"timeout=“100"> <context-id>http://../abcdef:012345<context-id/><context-id>http://../abcdef:012345<context-id/><context-service><context-service>http://.../HPSearch</ http://.../HPSearch</ context-context-

service>service>

<content>http://danube.ucs.indiana.edu:8080\x.xml</c<content>http://danube.ucs.indiana.edu:8080\x.xml</content>ontent>

</context></context>

shared state

<context xsd:type="ContextType"timeout=“100"><context xsd:type="ContextType"timeout=“100"><context-service><context-service>http://.../HPSearch</ http://.../HPSearch</ context-service>context-service><parent-context>http://../abcdef:012345<parent-context/><parent-context>http://../abcdef:012345<parent-context/><content> <content> shared data for HPSearch activity </content>shared data for HPSearch activity </content>

<activity-list mustUnderstand="true" mustPropagate="true"> <activity-list mustUnderstand="true" mustPropagate="true">

<service>http://.../DataFilter1</service> <service>http://.../DataFilter1</service>

<service>http://.../PICode</service> <service>http://.../PICode</service>

<service>http://.../DataFilter2</service> <service>http://.../DataFilter2</service>

</activity-list> </activity-list>

</context></context>

activity

3WMS

WFS

http://..../..../..txt

HP Search

Data Filter

PI Code

Data Filterhttp://..../..../tmp.xml

Context Information Service

4

7,8,9

10

6

5,11

WMS Client

Extended UDDI

0

1

2

The Interdependent Energy Infrastructure SimulationSystem (IEISS) in a Grid Environment

IEISS in a Grid environment:• IEISS provides modeling, simulation and analysis of interdependent

energy infrastructures. Input is data models in XML format. Los Alamos National Laboratory Argon National Laboratory

• Code location is unimportant, can be a service through remote execution• Results need to be stored, shared, annotated• Grid/Web Services can provide these capabilities

Problems:• How do we provide programming interfaces (not just user interfaces) to

the input data models?• How do we connect remote data sources directly to the IEISS code?

Solutions:• Use a SOA with GIS services to provide the input data, plot the output

data• Use WS-I compatible XML metadata services to support GIS workflow

applications by providing/maintaining static and dynamic information

WFS and WMS publish their WSDL URL to the UDDI Registry1-2-3 - WMS Client -> WMS Server -> UDDI -> WFS4-5 - WFS publishes the results as GML FeatureCollection document into a topic (“/NISAC/WFS”) in a pub/sub based messaging system. WFS -> WMS Server (creates a map overlay) and IEISS receive this GML document. WMS Server -> WMS Client (displays it)

6 - User invokes IEISS through WMS Client interface for the obtained geospatial features, and WMS Client starts a workflow session in the Context Service.

7 - On receiving invocation message, IEISS updates the shared state data to be “IEISS_IS_IN_PROGRES”. IEISS runs and produces an ESRI Shape file and then invokes shp2gml tool to convert produced Shape file to GML format. After the conversion IEISS updates shared session state to be “IEISS_COMPLETED”. As the state changes, the Context Service notifies all interested workflow entities such as WMS Client.

8 – On receiving the notification, WMS Client makes a request to the WFS-L for the IEISS output

9-10 - WFS-L publishes the IEISS output as a GML FeatureCollection document to NB topic ‘NISAC/WFS-L’. WMS Server is subscribed to this topic and receives the GML file then converts it to map overlay,and the Client displays the new model on the map.

Some ChallengesSome Challenges Security:Security: These applications have interesting security requirements These applications have interesting security requirements

• Authentication, authorization and priority for different usersAuthentication, authorization and priority for different users• Time/event dependent security for crisis response Time/event dependent security for crisis response

Performance:Performance: Are GIS services suitable for non-trivial data transfers? Are GIS services suitable for non-trivial data transfers?• We are using streaming data transfers for higher performanceWe are using streaming data transfers for higher performance• We research fast data transfer methodologies for higher performanceWe research fast data transfer methodologies for higher performance

Fault tolerance and scalability:Fault tolerance and scalability: We must support SOA based distributed We must support SOA based distributed application with reliable information sources.application with reliable information sources.• We design and build fault tolerant and high performance decentralized We design and build fault tolerant and high performance decentralized

Information Services to support metadata requirements of SOA.Information Services to support metadata requirements of SOA.

Adoption:Adoption: We must get the tools and services to the point where science We must get the tools and services to the point where science application developers want to use them application developers want to use them early early in the development process in the development process rather than rather than later.later.• Web Service client libraries to remote GIS dataWeb Service client libraries to remote GIS data• Develop codes to work with data streams rather than files.Develop codes to work with data streams rather than files.

Extensibility:Extensibility: This architecture can be enhanced by integrating various This architecture can be enhanced by integrating various types of distributed data sources that can provide geospatial data such as types of distributed data sources that can provide geospatial data such as MySQL, Oracle Spatial, ESRI SDE, Shape files etc.MySQL, Oracle Spatial, ESRI SDE, Shape files etc.

Location of information and software for Location of information and software for dependent Grid Projectsdependent Grid Projects

htpp://www.naradabrokering.orghtpp://www.naradabrokering.org provides Web service provides Web service (and JMS) compliant (and JMS) compliant distributed publish-subscribe distributed publish-subscribe messagingmessaging (software overlay network) (software overlay network)

http://www.crisisgrid.orghttp://www.crisisgrid.org is an OGC (open geospatial is an OGC (open geospatial consortium) Geographical Information System (GIS) consortium) Geographical Information System (GIS) compliant compliant GIS and Sensor Grid.GIS and Sensor Grid.

http://www.opengrids.orghttp://www.opengrids.org has WS-Context, Extended has WS-Context, Extended UDDI etc.UDDI etc.