astrogrid and ogsa-dai: early experiences
DESCRIPTION
AstroGrid and OGSA-DAI: Early experiences. Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge. Data Set Access: The DataCenter. DataCenter component. Data Access Plug-in. Web Services. Data. Tomcat / Axis. DataCenter JDBC plug-in. DataCenter component. JDBC. - PowerPoint PPT PresentationTRANSCRIPT
A PPARC funded project
AstroGrid and OGSA-DAI: Early experiences
Kona AndrewsGuy Rixon
Institute of Astronomy University of Cambridge
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Data Set Access: The DataCenter
DataCenter component
DataAccessPlug-in
Data
Web Services
Tomcat / Axis
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
DataCenter JDBC plug-in
DataCenter component
JDBC
DBMS
Web Services
Tomcat / Axis
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
DataCenter custom plug-in
DataCenter component
File querier
FITS files
Web Services
Tomcat / Axis
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
DataCenter proxy plug-in
DataCenter component
Service proxy
Vizier
Web Services
Tomcat / Axis
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
What we imagined
OGSA-DAI
Web Services
DataCenter Component
OGSA-DAI client plug-in
Tomcat / Axis
Grid Service Ports
Tomcat / OGSA
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Problem
OGSA-DAI
Web Services
DataCenter Component
OGSA-DAI client plug-in
Tomcat / Axis
Grid Service Ports
Tomcat / OGSA
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Problem
OGSA-DAI
Web Services
DataCenter Component
OGSA-DAI client plug-in
Tomcat / Axis
Grid Service Ports
Tomcat / OGSA
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Nasty solution
OGSA-DAI
Web Services
DataCenter Component
OGSA-DAI “client” plug-in
Tomcat / Axis
Grid Ports
Tomcat / OGSA
OGSA-DAIclient
Separate JVM
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMSADQL
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQLADQL
SQL
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
ADQL
SQL SQL
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
ADQL
SQL
Result rows
SQL
(to XML RowSet)
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
deliverFromURL
(to XML RowSet)
ADQL
SQL
Result rows
SQL
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
xslTransform
deliverFromURL
(to XML RowSet)
ADQL
SQL
Result rows
SQL
VOTable
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
xslTransform
deliverToFile / deliverToGFTP
deliverFromURL
(to XML RowSet)
ADQL
SQL
VOTable
Result rows
SQL
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Current use of OGSA-DAI activities
DataCenter
OGSA-DAI
PostgresDBMS
ADQL to
SQL
sqlQueryStatement
xslTransform
deliverToFile / deliverToGFTP
deliverFromURL
(to XML RowSet)
ADQL
SQL
VOTable
Result rows
SQL
Deliver results
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Comments
Astronomers often need LARGE result sets
Thousands to millions of rows
Memory usage is a big issue
OGSA-DAI 3.1 much better than 3.0
xslTransform activity hogging memory
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
OGSA-DAI speed penalty
0
10
20
30
40
1e3 1e4 1e5 2e5 4e5 1e6 2e6
Number of result rows
Rough figures only
Measured without xslTransform activity
Will likely vary with available memory
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Planned extension: User tables
OGSA-DAI
DBMSDataCenter
R/ODatasets
Query Query results
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Planned extension: User tables
OGSA-DAI
DBMSDataCenter
R/ODatasets
Usertables
Query Query results Data flow
GDS to GDS transfer
Bulk load
OGSA-DAI
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
DBMS
Planned extension: User tables
OGSA-DAI
DBMSDataCenter
DataCenter
OGSA-DAI
Table Manager
R/ODatasets
Usertables
Query Query results Data flow
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
DBMS
Planned extension: User tables
OGSA-DAI
DBMSDataCenter
DataCenter
OGSA-DAI
Table Manager
Client
User data
R/ODatasets
Usertables
Query Query results Data flow
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Comments
Will be relying on sqlBulkLoadRowSet (!)
Table management facilities currently thin
Not using X.509 auth at user granularity
Still require W/S-based access to G/S
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
Our needs
Ability to handle very large result sets - Don’t rely on loading whole result set into memory!
Suite of DB/table management activities- List tables in DB
- Describe table column types / space usage
- Last access/modification time
- Etc etc etc
Ability to create and delete indices on tables
Ultra-simple installation procedures
OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences 07-04-2004
The End
www.astrogrid.org