grid on demand
DESCRIPTION
Slides of the Presentation (30 july 2010) of Grid on DemandTRANSCRIPT
![Page 1: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/1.jpg)
Welcome
Grid on Demand
Willem Toorop and Alain van Hoof
{wtoorop,ahoof}@os3.nl
June 30, 2010
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 1 / 39
![Page 2: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/2.jpg)
Introduction Research Question
Research Question
Can Grid computing be offered as a Cloud service and whattype of applications benefit from this?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 2 / 39
![Page 3: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/3.jpg)
Introduction The Grid and The Cloud
The Grid and The Cloud: Sharing Resources
Sharing Resources to minimizing idle resources
Resource Capacity GoodsProcessor Flops Compute
Hard-disk memory Bytes Storage
Network Bytes/sec Communication Bandwidth
Grid Cloud (IaaS)Distribution By Collaboration By Cloud Provider
Resource Type Heterogeneous Homogeneous
Abstraction Middleware (open standards) Virtualization
Sharing Guarantee Security infrastructure Financial infra
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 3 / 39
![Page 4: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/4.jpg)
Introduction The Grid and The Cloud
The Grid and The Cloud: Dividing the Resources
Cloud
Grid
GridGrid
User
User
User
VOUser User
VOUser User
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 4 / 39
![Page 5: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/5.jpg)
A Grid in the Cloud Urgent Computing
A Grid in The Cloud: Urgent Computing
GRIDon
Demand
Using the elasticity and scalability of the Cloud and providing a Gridinterface, current e-Science (computationally intensive science)applications can use a Grid on Demand when there is a high demand forresources for a short period of time.
Example
UrbanFlood (http://urbanflood.eu) wants to calculate flooding predictionswhen actual flooding happens, Grid on Demand can provide this.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 5 / 39
![Page 6: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/6.jpg)
A Grid in the Cloud Grid on Demand
Grid on Demand
Compute Nodes
Head Node
Job Submission Client
OS: Ubuntu 10.04Grid: Globus 4.2.1PBS: Torque 2.6.8
Grid Interface: Globus Toolkit 4.2 (open standard, widely used)Queue Manager (Distribution of jobs to nodes): Torque 2.6.2Amazon EC2 Cloud Provider (de-facto standard with open API)
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 6 / 39
![Page 7: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/7.jpg)
Implementation What makes the Grid, the Grid
What makes the Grid, the GridTypical Grid usage example spanning multiple organizations
GridUser
Scheduler Storage
Compute
Compute
Organization A
Organization B
Organization D
Organization C
store data
use my datato compute X
use 1st 1
2of data
to compute12X
use 2nd 12 of data
to compute 12 X
accessdata
access data
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 7 / 39
![Page 8: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/8.jpg)
Implementation What makes the Grid, the Grid
Proxy certificates for delegation
The Grid is a collaboration between multiple organizations.
Grid users need to be authorized to use remote resources
Those resources should be authorized to perform actionson behalf of the Grid user
Proxy certificates enable delegation of authority
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 8 / 39
![Page 9: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/9.jpg)
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
![Page 10: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/10.jpg)
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute XVirtual Organizations are based ondelegation by proxy certificates.
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
![Page 11: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/11.jpg)
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
![Page 12: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/12.jpg)
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Fixed CA in image?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
![Page 13: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/13.jpg)
Implementation What makes the Grid, the Grid
Grid user
create proxy Scheduler
create cert.request
Compute
compute X
sign this
Okay
compute X
Proxy certificates allow fordynamic resource allocation,
But the hosts still need aCA signed certificate.
This is a problem when hostname’sare assigned dynamically!
Fixed CA in image?
Dynamic CA on every Launch?
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 9 / 39
![Page 14: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/14.jpg)
Implementation Amazon Elastic Block Store
Easy Image management with Amazon Elastic Block Store
(A,S) PendingRunning
(IA,VS ,H)
Stopping
Stopped(IA,VS)
Pending
Shutting-down
TerminatedIA
(A′,S ′)
launchinstance IA
VS ← S
stop
star
t
terminate
terminate
create AMI A′
S ′ ← VS
configure
Life cycle of Amazon Machine Images using Elastic Block StorageA = Amazon Machine Image, S = Snapshot,I = Instance, V = Volume, H = Hostname
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 10 / 39
![Page 15: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/15.jpg)
Implementation EBS for configuration stages
EBS for configuration stages
(A, S)Running
(IA,VS ,H)
Stopped(IA,VS)
(A′, S ′)
Stopped(IA,VS)
(A′′, S ′′)
Running(IA,VS ,H)
Stopped(IA,VS)
(A′′′, S ′′′)
launchinstance IA
VS ← S
shu
tdow
n
forreu
se star
t
create AMI A′
S ′ ← VS
create AMI A′′
S ′′ ← VS
create AMI A′′′
S ′′′ ← VSshutdow
n
forsharing
shut
dow
nfo
rsh
arin
g shu
tdow
n
forreu
se star
t
provide Amazon Access Credentials
(configure cluster)
launch IA′
VS ← S ′launch IA′′′
VS ← S ′′′
configure configure
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 11 / 39
![Page 16: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/16.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 12 / 39
![Page 17: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/17.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 13 / 39
![Page 18: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/18.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 14 / 39
![Page 19: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/19.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 15 / 39
![Page 20: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/20.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 16 / 39
![Page 21: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/21.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 17 / 39
![Page 22: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/22.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 18 / 39
![Page 23: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/23.jpg)
Web Interface Launching Grid on Demand
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 19 / 39
![Page 24: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/24.jpg)
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 20 / 39
![Page 25: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/25.jpg)
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 21 / 39
![Page 26: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/26.jpg)
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 22 / 39
![Page 27: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/27.jpg)
Web Interface Configuring the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 23 / 39
![Page 28: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/28.jpg)
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 24 / 39
![Page 29: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/29.jpg)
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 25 / 39
![Page 30: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/30.jpg)
Web Interface Real Grid certificate usage
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 26 / 39
![Page 31: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/31.jpg)
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 27 / 39
![Page 32: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/32.jpg)
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 28 / 39
![Page 33: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/33.jpg)
Web Interface Elasticity of the cluster
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 29 / 39
![Page 34: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/34.jpg)
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 30 / 39
![Page 35: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/35.jpg)
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 31 / 39
![Page 36: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/36.jpg)
Web Interface Saving your settings
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 32 / 39
![Page 37: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/37.jpg)
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
![Page 38: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/38.jpg)
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
![Page 39: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/39.jpg)
“Real use” and Performance WS-VLAM and WAVE
“Real use” and Performance test
To test the possibilities and performance of Grid on Demand a BiomedicalApplication and a workflow manager are used
Biomedical Application: WAVE
Parallel (MonteCarlo simulation) application
Workflow Manager
A (graphical) tool to assist complex e-Science application creation
Creates a series of jobsubmissions
WS-VLAM created by UvA SNE Group
Connects to Globus Grid Interface
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 33 / 39
![Page 40: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/40.jpg)
“Real use” and Performance Connecting the Grid on Demand to a Workflow Manager
Connecting the Grid on Demand to a Workflow Manager
Compute Nodes
Head Node
OS: Ubuntu 10.04Grid: Globus 4.2.1PBS: Torque 2.6.8
WAVE Biomedical Compute
Application
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 34 / 39
![Page 41: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/41.jpg)
“Real use” and Performance Preliminary performance results: total execution time
Preliminary performance results: total execution time
Grid on demand compared to a 32 node physical UvA cluster (DAS3) withGlobus Grid interface
Grid Jobs Total execution timeDAS3 28 00:42:58
Grid on Demand 28 00:46:22
DAS3 98 02:46:36
Grid on Demand 98 00:52:26
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 35 / 39
![Page 42: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/42.jpg)
“Real use” and Performance Preliminary performance results: 30 Jobs
Preliminary performance results of job pending andrunning time: 30 jobs
00:00:00
00:10:00
00:20:00
00:30:00
00:40:00
00:50:00
01:00:00
01:10:00
01:20:00
0 5 10 15 20 25 30
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on DAS3 - 30 Jobs
Running timePending time
00:00:00
00:10:00
00:20:00
00:30:00
00:40:00
00:50:00
01:00:00
01:10:00
01:20:00
0 5 10 15 20 25 30
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on Grid on Demand 30 Jobs
Running timePending time
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 36 / 39
![Page 43: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/43.jpg)
“Real use” and Performance Preliminary performance results: 100 Jobs
Preliminary performance results of job pending andrunning time: 100 jobs
00:00:00
00:15:00
00:30:00
00:45:00
01:00:00
01:15:00
01:30:00
01:45:00
02:00:00
02:15:00
02:30:00
0 10 20 30 40 50 60 70 80 90 100
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on DAS3 - 100 Jobs
Running timePending time
00:00:00
00:15:00
00:30:00
00:45:00
01:00:00
01:15:00
01:30:00
01:45:00
02:00:00
02:15:00
02:30:00
0 20 40 60 80 100
Tim
e (
HH
:MM
:SS
)
Job Number
Performance on Grid on Demand - 100 Jobs
Running timePending time
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 37 / 39
![Page 44: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/44.jpg)
Conclusion
Conclusion
Grid on Demand can offer Grid as a Cloud service
Parallel applications run/scale very well on Grid on Demand
Grid on Demand can be a solution for Urgent Computing demands
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 38 / 39
![Page 45: Grid on Demand](https://reader036.vdocuments.mx/reader036/viewer/2022081400/555087d0b4c905a85c8b4b00/html5/thumbnails/45.jpg)
The End
Questions
Willem Toorop and Alain van Hoof (OS3) Grid on Demand June 30, 2010 39 / 39