a pparc funded project astrogrid grid data warehouse iteration 5 workgroup meeting
TRANSCRIPT
A PPARC funded project
AstroGrid Grid Data Warehouse
Iteration 5 workgroup meeting
12-02-2004 GDW It05 Meeting 2
End of It4.1: Ideal view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
12-02-2004 GDW It05 Meeting 3
End of It4.1: Ideal view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
HTTP
12-02-2004 GDW It05 Meeting 4
End of It4.1: Ideal view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
Web ServiceHTTP
12-02-2004 GDW It05 Meeting 5
End of It4.1: Ideal view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
Web ServiceHTTP
Grid Service
12-02-2004 GDW It05 Meeting 6
End of It4.1: Actual view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
Grid Service
For testing
12-02-2004 GDW It05 Meeting 7
End of It4.1: Actual view
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
Grid Service
For testing
GDWClient
Shell to JVM
12-02-2004 GDW It05 Meeting 8
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Not tested at this level
12-02-2004 GDW It05 Meeting 9
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Hard to write ADQL/XML
Not tested at this level
12-02-2004 GDW It05 Meeting 10
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Data flow inefficiencies
Hard to write ADQL/XML
Not tested at this level
12-02-2004 GDW It05 Meeting 11
Performance and portability issues
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Data flow inefficiencies
Hard to write ADQL/XML
Not tested at this level
12-02-2004 GDW It05 Meeting 12
Performance and portability issues
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Data flow inefficiencies
Hard to write ADQL/XML
Re-indexing, diskspace issues
Not tested at this level
12-02-2004 GDW It05 Meeting 13
Performance and portability issues
End of It4.1: Issues
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Performance issues, data size limits
Data flow inefficiencies
Hard to write ADQL/XML
Re-indexing, diskspace issues
Not tested at this level
12-02-2004 GDW It05 Meeting 14
Aside: Future issue
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
12-02-2004 GDW It05 Meeting 15
Datacenter
Iteration 5: New functionality
Portal JES MySpace
OGSA-DAI
PostgreSQL
Browser
GDWClient
Grid FTP data channel
Control channel
GDW
12-02-2004 GDW It05 Meeting 16
Iteration 5: New functionality?
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Grid FTP data channel direct to MySpace
Control channel
12-02-2004 GDW It05 Meeting 17
Iteration 5: Infrastructure updates
Upgrade to OGSA-DAI 3.1
Speed improvements
Scalability improvements
Stability improvements
Improved support for large result sets
12-02-2004 GDW It05 Meeting 18
Iteration 5: Infrastructure updates
Reload of GDW datasets in PostgreSQL
Re-index for efficient querying
Recover disk space
Upgrade PostgreSQL on cass123?
12-02-2004 GDW It05 Meeting 19
Performance and portability issues
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
Performance issues, data size limits
Data flow issues; no async queries
Hard to write ADQL/XML
Re-indexing, diskspace issues
Not tested at this level
12-02-2004 GDW It05 Meeting 20
Performance and portability issues
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
OGSA-DAI upgrade (?)
Data flow issues; no async queries
Hard to write ADQL/XML
Re-indexing, diskspace issues
Not tested at this level
12-02-2004 GDW It05 Meeting 21
Performance and portability issues
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
GDWClient
OGSA-DAI upgrade (?)
Data flow issues; no async queries
Hard to write ADQL/XML
DBMS reload
Not tested at this level
12-02-2004 GDW It05 Meeting 22
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
OGSA-DAI upgrade (?)
Data flow issues; no async queries
Hard to write ADQL/XML
DBMS reload
Not tested at this level
OGSA-DAI upgrade (?)
12-02-2004 GDW It05 Meeting 23
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
OGSA-DAI upgrade (?)
Hard to write ADQL/XML
DBMS reload
Not tested at this level
OGSA-DAI upgrade (?)
Refactor with streams; async queries
12-02-2004 GDW It05 Meeting 24
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
OGSA-DAI upgrade (?)
DBMS reload
Not tested at this level
OGSA-DAI upgrade (?)
Refactor with streams; async queries
ADQL/TEXT Query Builder
12-02-2004 GDW It05 Meeting 25
Iteration 5: Enhancements
Portal JES MySpace
OGSA-DAI
PostgreSQL
Datacenter
GDW
Browser
OGSA-DAI upgrade (?)
DBMS reload
OGSA-DAI upgrade (?)
Refactor with streams; async queries
ADQL/TEXT Query Builder
Continuous integration
12-02-2004 GDW It05 Meeting 26
Future direction
Browser Portal
Datacenter
OGSA-DAI
PostgreSQL
OGSA-DAI
DBMS
W/H
Warehouse /MySpace(T)
D/C
JDBCJob manager
VOTable
“Bilingual”GDS-GDS link
Compute-grid stuff
Read/write Read-only
MySpace(F)JES
12-02-2004 GDW It05 Meeting 27
Continuous Integration
All assigned programming tasks have a bugzilla entry
All non-trivial changes are made in a new branch
The new branch is compiled and integration-tested before being remerged
Appropriate unit tests should be supplied
12-02-2004 GDW It05 Meeting 28
Useful resources
Iteration 5 wiki page: Iteration5
GDW Iteration 5 wiki page: Itn05Grid
Please add useful links to our wiki page
12-02-2004 GDW It05 Meeting 29
The End
12-02-2004 GDW It05 Meeting 30
Task List (1)
TASKS (week 3) KEA: Upgrade OGSA-DAI ECA: On holiday CGP: Reload datasets; host cert for hydra
TASKS (weeks 4-5) KEA: Refactor querier client for OGSA-DAI
3.1 ECA: Produce Postgres-optimised ADQL-SQL
translator (for ADQL 0.5) CGP: Assist ECA; sample ADQL queries
12-02-2004 GDW It05 Meeting 31
Task List (2)
TASKS (week 6-7) KEA: GridFTP transfers from OGSA-DAI to DC ECA: Asynchronous OGSA-DAI queries CGP: GDW install script, query performance
TASKS (week 8) Code freeze 26/03/04 Final testing, packaging, documenting Planning for Iteration 6