On-Demand Virtual Workspaces: Quality of

Life in the Grid

Kate Keaheykeahey@mcs.anl.gov

Argonne National Laboratory

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

the Grid metaphor

How do we store energy?

How do we charge for energy?

How do we reliably deliver energy?

What happens if a power station fails?

How do we ensure quality of service?

What elements make for a safe and efficient power Grid?

How do we make sure that supply meets demand?

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

computational Grids

How do we store computing?

How do we charge for computing?

How do we reliably deliver cycles?

What happens if a power station fails?

How do we ensure quality of service?

What elements make for a safe and efficient power Grid?

How do we make sure that supply meets demand?

What is the “unit” of resource usage?

How can we manage different computing environments?

How can we ensure that disk, CPUs, network are all available?

How can we negotiate for computation?

NCSA

ANLCaltech

SDSC

Tera Grid

Grid Middleware

We need a vehicle that will enable usto use Grid resources as easily and

intuitively as we use electrical power today

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

what is virtualization?

Let’s see what’s availableand adapt my problem

to use it

Here is the environment I need to solve my problem-- deploy it on the Grid

Can we provide the middleware that will enable this change of approach?

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

virtual workspaces

Focus on execution environments Two aspects of workspaces:

Environment definition: We get exactly the (software) environment me need on demand.

Resource allocation: Provision and guarantee all the resources the workspace needs to function correctly (CPU, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions.

Environment and resource allocation are now independent

Qua

lity

ofLife

Qua

lity

of

Serv

ice

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

how can we implement VWs?

Configuring physical machines Slow and invasive Environments are hard to describe Limited/none enforcement options Using environment management tools

Virtual Machines Fast to deploy, much less invasive Environments are easy to describe Bonus: isolation, serialize, redeploy, migrate

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

virtual machine primer

Hardware

Virtual Machine Monitor (VMM) / Hypervisor

Guest OS(Linux)

Guest OS(NetBSD)

Guest OS(Windows)

VM VM VM

AppApp AppAppApp

Xen

VMWare

UML

Denali

etc.

Paravirtualization makes the performance overhead very acceptable

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

virtualizing other elements of an environment

Virtual storage Combining many distributed physical

resources Virtual networks

Namespace management Virtual private networks, ViNE, virtuoso, VIOLIN

Quality of Service Overlay networks

Toward Virtual Grids Putting all these elements together

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

GT4 workspace service

The GT4 Virtual Workspace Service (VWS) allows an authorized client to deploy and manage workspaces on-demand. GT4 WSRF front-end Leverages multiple GT services Currently implements workspaces as VMs

Uses the Xen VMM but others could also be used

Current release 1.2 (September, 06) http://workspace.globus.org

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

workspace service backstage

Poolnode

Trusted Computing Base (TCB)

ImageNode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

The workspace service has a WSRF frontend that allows users to deploy and manage

virtual workspaces

The VWS manages a set of nodesinside the TCB (typically a cluster).

This is called the node pool.

Each node must have a VMM (Xen)installed, along with the workspacebackend (software that manages

individual nodes)

VM images are staged to adesignated image node

inside the TCB

VWSNode

VWSService

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

ImageNode

deploying workspaces

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Workspace

- Workspace metadata (with image location)- Deployment request

VWSService

Adapter-based implementation model Transport adapters

Default scp, then gridftp

Control adapters Default ssh Deprecated: PBS, SLURM

VW deployment adapter Xen Previous versions: VMware

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

workspace request arguments

A workspace, composed of: VM image Workspace metadata

XML document Includes deployment-independent information:

VMM and kernel requirements NICs + IP configuratoin VM image location

Need not change between deployments Resource allocation

Specifies availability, memory, CPU%, disk Changes during or between deployments

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

ImageNode

interacting with workspaces

Poolnode

Trusted Computing Base (TCB)

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

Poolnode

The workspace service publishesinformation on each workspace

as standard WSRF ResourceProperties.

Users can query thoseproperties to find out

information about theirworkspace (e.g. what IP

the workspace was bound to)

Users can interact directly with their

workspaces the same way the would with a

physical machine.

VWSService

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

workspace service interfaces

Workspace Service

Workspace FactoryService

Create()

Workspace Meta-data/Image

Deployment Request

inspect & manage

notify

Workspace Resource Instance

authorize & instantiate

Workspace Service

Handles creation of workspaces.Also publishes information onwhat types of workspaces it

can support

Handles management ofeach created workspace

(start, stop, pause, migrate,inspecting VW state, ...)

Resource Properties publish theassigned resource allocation, how VW was bound to metadata (e.g.IP address), duration, and state

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

status Latest Release: 1.2 released 9/14

Significant improvement over 1.1.1 At least one more release planned by the end of the

year to include C client and better IP handling among others

To be included in the next VDT release VW is an incubator project in dev.globus

New governance model for Globus Toolkit http://dev.globus.org All software released under Apache license 2.0

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

support

And that’s what we do to bugs!

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

applications: ESF

CDF

CMS ATLAS

Guest VO

ESF

SE CE

Site

GT4 Workspace Service & VMM

Dynamically deployed ES Wafers for each VO

Wafer images stored in SE

Compute nodes and Storage nodes

www.opensciencegrid.org/esf

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

ESF: division of laborComparison of Request Throughput over Time

0

5

10

15

20

25

30

30 90 150 210 270 330 390 450 510 570 630 690 750 810

Time (in 30 second buckets)

Completed jobs

VO1Client VO2Client

VO2 (under changing VO1 load conditions

0

2

4

6

8

10

12

14

16

30 60 90 120 150 180 210 240 270 300 330 360 390 420 450 480

Time (in 30 second buckets)

Jobs completed

1mill-VO2 2mill-VO2 3mill-VO2

Paper: “Division of Labor: Tools for Growth and Scalability of Grids”, ICSOC 2006

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

no STAR

applications: STAR

no STAR

no STAR

STAR

STAR

STAR

VWS

GRAM

STAR

Provisioning STAR nodes on TeraPort (UC): demonstrated at SC06 show floor

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

are we there yet?

YES: we do have reliable infrastructure that can implement the basic virtualization scenario

NO: the basic scenario addresses about 10% of virtualization potential (on a good day)

Yes.

And No...

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

a chicken and egg problem

Chicken Egg

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

meet the chicken Overcoming Xenophobia

Hypervisor installations are “invasive” We need flexible site resource management systems

Security: the cure or the disease? On the whole the cure, but it is a new tool

Will it scale? This is not a question that a simulation could answer!

We need more effort in this area Commercial deployments are moving faster

Hosting services, Amazon’s EC2, others… There are more incentives

Pioneering is hard! OSG

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

meet the egg Suppose you have this infrastructure deployed, now

what? Where would be iTunes without music?

Original idea: develop a library of VM images Labor intensive Images “age”

“Assembly line” approach rPath: scientific appliances and rBuilder

Appliance = application + its environment

BCFG2: configuration management tool Producing and managing images

How do we describe, indentify, and query to find the right image?

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

virtualizing clusters

How do we construct virtual clusters? How do we deploy virtual clusters on hardware

resources? (overcoming xenophobia) The overhead should be “invisible” to the client Can we take advantage of application-specific knowledge

when we schedule VMs? What scheduler logic is appropriate and needed for

scheduling workspaces?

Papers “Virtual Clusters for Grid Communities”, CCGrid06 “Overhead Matters: A Model for Virtual Resource Management”, VTDC 2006

(in SC06)

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

toward virtual grids

Deploying workspaces across multiple sites Remember the STAR application

Virtualizing multiple aspects of a Grid Combining networking and storage

Use Case: Combining QoS on data movement and execution We want to get rid of workspace staging!

These are good times to be a meta-scheduler!

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

details, details…

Looking down the road Assume we have resolved the “simple”

problem… What if we succeed?

100s of VMs per physical manchine Name management, storage, etc.

On the bright side There may also be pleasant surprises

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

conclusions

We live in exciting times! Making progress is hard We have useful infrastructure that is being

used by projects today on a small scale -- we need to move to larger scales

There are still many open problems We have work to do!

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

Virtualization Workshop

Virtualization Technology in Distributed Computing (VTDC) 2006 Co-held with SC06 http://workspace.globus.org/vtdc2006

Virtual Workspaces 5th Meeting of Spanish Initiative in Grid Middleware

credits

Workspace team Tim Freeman Borja Sotomayor

With guest appearances by: Ian Foster, Elizeu Santos-Neto, Frank

Siebenlist, and others