OGSA-DAI Open Grid Services Architecture – Data Access and Integration

NeSC Review

18 March 2004

Description and Aims

OGSA–DAIProvide a uniform access framework for heterogeneous data resources on the GridData resources:

Relational databases XML collections Can widen scope, e.g. files, any data source/sink

Middleware which:Reduces development cost of data centric Grid applicationsFacilitates Grid data centric application developmentFacilitates data integrationIncreased collaboration

Status

Functional Scope defined goalsHad a 66 point functional scope for Phase IIAll MUSTs/SHOULDs achieved except: Statement Metadata (partially complete)

– We do not list all SQL operations a GDS supports

MAYs completed: Transforming application data (compression, XSLT) Caching of data Block transfer of data Scripting (though we no longer call it that) Persistent components

Workplan

All workpackages and deliverables completedThe project lasted 92 (78 + 14) weeks.Over the 92 weeks (4/2/02 to 31/10/03):

138 SM of effort at EPCC were planned140.2 SM were expended1% overrun

Workpackage (WP) Status

1. Programme Management 1.1 End of Phase 2 review D1.1 End of Phase 2 Report

FINISHED OracleComplete

2. Architecture 2.1 High Level Design D2.1 High level system architecture Due M3, 31/12/2002, delivered 25/11/2002 2.2 Architectural definition D2.2 Architecture framework definition Due M3, 31/12/2002, delivered 7/11/2002

FINISHEDEPCC/IBMCompleteEPCC/IBM/NeSC/UoMComplete

3. Development 3.1 Detailed Component design D3.1.1 Release design documentation (Release 1) D3.1.2 Release design documentation (Release 2) D3.1.3 Release design documentation (Release 3)3.2 Component implementation D3.2.1 Release implementation code (Release 1) D3.2.2 Release implementation code (Release 2) D3.2.3 Release implementation code (Release 3)3.3 Write documentation D3.3.1 Release system and user documentation (Release 1) D3.3.2 Release system and user documentation (Release 2) D3.3.3 Release system and user documentation (Release 3)3.4 Develop test strategy and test suites D3.4 Release test strategy and test suites3.5 Perform tests D3.5.1 Documented test results (Release 1) D3.5.2 Documented test results (Release 2) D3.5.3 Documented test results (Release 3)

FINISHEDEPCC/IBMCompleteCompleteCompleteEPCC/IBMCompleteCompleteCompleteEPCC/IBMCompleteCompleteCompleteEPCC/IBMCompleteEPCC/IBMCompleteCompleteComplete

4. Distributed Query Processing4.1 Revise DQP model D4.1 Revised DQP model4.2 Develop prototype D4.2 Distributed query prototype

FINISHEDUoM/UoNCompleteUoM/UoNComplete

5. Release Management Complete (Phase 2)

Releases

Releases added functionality in staged deliveriesKept on target

Max slippage was 2 weeks due to GGF

Made available through project website and GTR

http://www.ogsadai.org.ukhttp://gtr.globus.org

Early adopters had early access to release candidates1199 downloads at 31st Oct 2003

50% from UK

Release Release Date

Release 3.0.2 26/09/03

Release 3 31/07/03

Release 2 interim

11/06/03

Release 2 15/04/03

Release 1 interim

28/02/03

Release 1 15/01/03

The Basics

Data Resource

Container

DAISGR

Client GDSF

GDS

Technical Achievements

Grid Data ServicePerform documents allow for powerful “scripting”

Composition of requests (encapsulation of activities)

Activity Framework easily extended by developersVariety of delivery/upload mechanisms

SOAP/HTTP, GridFTP, GDT

Can achieve complex composition patterns e.g. distributed queries using temporary tables

Grid Data Service FactorySimple to configureSupported databases:

MySQL, DB2, Oracle, XIndice

Other “working” databases: SQL Server, Postgres, Access (via JDBC/ODBC)

DAI Service Group RegistryFramework for service discovery

Dissemination (1)

Selected Presentations (EPCC during Phase II only)10th Anniversary of Poznan Supercomputing Centre, October 24, 2003Designing and Building Grid Services Workshop, Chicago, October 8, 2003Glasgow Kelvin Hub opening, September 17, 2003All Hands presentations and demonstrations, Nottingham, September 2-4, 2003DAIS F2F, NeSC, August 21 – 22, 2003ASTAR Visit, NeSC, July 14, 2003Virtual Observatory as a Data Grid, NeSC, June 30 – July 2, 2003Geoffrey Fox visit, EPCC, April 4, 2003NeSC Review, NeSC, March 28, 2003OGSA-DAI / Informatics meeting, NeSC, March 27, 2003OGSA Experiences Panel, GGF7 Tokyo, March 4-7, 2003NeSC Open Day, NeSC, January 17, 2003

Dissemination (2)

PostersNeSC Review, NeSC, September 30, 2003UK e-Science All Hands, Nottingham, September 2-4, 2003 GlobusWorld January 13 – 17, 2003

Publications through GGFDAIS – File Access, September 19, 2003DAIS – Grid Data Service Specification, September 19, 2003DAIS – Relational Specialisation, September 19, 2003DAIS – XML Specialisation, September 19, 2003DFDL – Basic Structures Ontology, August 5, 2003DFDL – Primitive Type Ontology, August 5, 2003DFDL – Structural Description, August 5, 2003DFDL – XML Representation, August 5, 2003DFDL – Primer, June 4, 2003

Other notable publications“Grid Security for Dummies”, available from OGSA-DAI website, October 29, 2003

Training

Course and tutorials by EPCC staff were run at:eScience Summer School, NeSC, September 29 – October 3rd, 2003International Summer School on Grid Computing, Naples, July 13 – 25, 2003OGSA-DAI Training Course, NeSC, April 22, 2003OGSA-DAI Tutorial, GGF7 Tokyo, March 4, 2003 Creating Grid Services using GT3 and Java course, NeSC, February 24, 2003OGSA-DAI Training Course, NeSC, February 11, 2003OGSA-DAI Training Course, NeSC, January 8, 2003

“Show and tell” method of increasing exposure

Support

Support for OGSA-DAI through Grid Support Centre from Release 2

Very useful to encourage user take-up

Query deskRegular stream of queries

Active user list“Power Users” submitted answers to other users questionsDiscussed innovative ways of extending OGSA-DAI

Exploitation

Projects started at both EPCC/NeSC and IBM using OGSA-DAI:

eDiaMoNDFirstDIGINWABRIDGESEdSkyQueryG

Many more projects using OGSA-DAIPresentations, Visits and Training

have been vital to uptake of OGSA-DAI

Future plans

Work continues under the DAIT projectResearch and develop OGSA-DAI softwareImprove performance and scalabilityLiase with technology adopters

Make sure OGSA-DAI works for them

Liase with Globus Globus Alliance OGSA-DAI also distributed through Globus Toolkit

Continue standardisation process through DAIS

We’ve done Data Access… now it’s time for Data Integration!

Project Participants

EPCCAli Anjomshoaa, Mario Antonioletti, Rob Baxter, Neil Chue Hong, Ally Hume, Mike Jackson, Amy Krause, Jeremy Nowell, Charaka Palansuriya, Tom Sugden, Martin Westhead

IBM UKBrian Collins, Simon Laws, Andrew Borley, James Magowan, Neil Hardman, George Hicken, Manfred Oevers, Alan Knox

IBM USSusan Malaika, Inderpal Narang

NeSCMalcolm Atkinson

Oracle UKDave Pearson

University of ManchesterNorman Paton, Nedim Alpdemir

University of NewcastlePaul Watson, Arijit Mukherjee