Introduction to OGSA-DAI

Neil Chue HongOGSA-DAI Project Manager

14th February 2006GGF16, Athens

GGF16, Feb 2006 ©

2

Data Services: challenges Scale

Many sites, large collections, many uses Longevity

Research requirements outlive technical decisions Diversity

No “one size fits all” solutions will work Primary Data, Data Products, Meta Data, Administrative data, …

Many Data Resources Independently owned & managed Geographically distributed

and I haven’t even mentioned security yet!

GGF16, Feb 2006 ©

3

Use Cases for Data Services Data Filtering:

Single source producing large amounts of data distributed to many sites downstream

Data Discovery: many sources, many query entry points in a linked system

Data Translation: source to sink, conversion of data model / structure

Data Federation: many sources, linked to provide view as a single source

Data Replication full or partial copies to improve throughput

Data Integration (model aggregation) e.g. integration of time variant data, streams, files

Data Integration (knowledge expansion) forming links between databases to increase knowledge

GGF16, Feb 2006 ©

5

Requirements on Data Services? Common Data Model e.g. RowSet

Common Query Language(s) e.g. XQuery, SQL

Standard access to data resource schema information physical data resource information for optimisation purposes data resource descriptive information for discovery / integration

Single, seamless security model

Dynamic publication and discovery

Multiple, efficient delivery methods

Move computation towards data

Data aggregation functionality

Replication information

GGF16, Feb 2006 ©

6

OGSA-DAI In One Slide An engineered extensible framework

for data access and integration. Expose heterogeneous data

resources to a grid through web services.

Interact with data resources: Queries and updates. Data transformation / compression Data delivery.

Customise for your project using Additional Activities Client Toolkit APIs Data Resource handlers

A base for higher-level services federation, mining, visualisation,…

GGF16, Feb 2006 ©

7

MySQL

OGSA-DAI service

Engine

SQLQuery

JDBCData

Resources

Activities

DB2

GZip GridFTPXPath

XMLDB

XIndice

readFile

File

SWISSPROT

XSLT

SQLServer

Data-bases

ApplicationApplicationClient ToolkitClient Toolkit

GGF16, Feb 2006 ©

8

MySQL

OGSA-DAI service

Engine

SQLQuery

JDBC

SQL

JDBC

SQL

JDBC

SQL

JDBC

SQL

JDBC

MultipleSQL GDS

SQLQuery

GGF16, Feb 2006 ©

9

Distributed Query Processing Higher level services

building on OGSA-DAI Queries mapped to

algebraic expressions for evaluation

Parallelism represented by partitioning queries Use exchange operators

table_scan(protein)

table_scantermID=S92(proteinTerm)

reduce

reduce

hash_join(proteinId)

op_call(Blast)

reduce

exchange

exchange

3,4

1 2

GGF16, Feb 2006 ©

10

DQP architecture

Co-ordinator

Evaluator Evaluator Evaluator

OGSA-DAI

OGSA-DAI

OGSA-DAI

OGSA-DAI

Query SQL & OQL

OGSA-DAI activity

WS-I only

Using client toolkit

All interfaces that aresupported by toolkit

GGF16, Feb 2006 ©

11

Contributing to OGSA-DAI

Additional functionality: Provide activities which implement specific

functionality Provide extra client functionality Provide different security mechanisms Provide higher level components and applications

Different levels of contributions Based on OGSA-DAI? Works with OGSA-DAI? Part of OGSA-DAI?

GGF16, Feb 2006 ©

12

Future plans A new version of the OGSA-DAI Engine

better support for concurrency, sessions, monitoring and notification

Implementing new DAIS specifications Key things that we will be addressing:

Performance (particularly format representation and transport) Security Model which can be applied across platforms Transactions provision More data integration facilities

Integration with other components registries (e.g. GRIMOIRES) workflow editors (e.g. Taverna)

Working with new projects e.g. CancerGrid, iSpider, GEODE

GGF16, Feb 2006 ©

13

Further information The OGSA-DAI Project Site:

http://www.ogsadai.org.uk The DAIS-WG site:

http://forge.gridforum.org/projects/dais-wg/ OGSA-DAI Users Mailing list

users@ogsadai.org.uk General discussion on grid DAI matters

Formal support for OGSA-DAI releases http://bugs.ogsadai.org.uk

OGSA-DAI training courses