ppt
Post on 27-Nov-2014
1.700 Views
Preview:
DESCRIPTION
TRANSCRIPT
Cloud Computing with Nimbus April 2009
Kate Keahey
(keahey@mcs.anl.gov)
University of Chicago
Argonne National Laboratory
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Cloud Computing
IaaSInfrastructure-as-a-Service
PaaSPlatform-as-a-Service
SaaSSoftware-as-a-Service
elasticitycomputing on demand
capital expense
operational expense
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Cloud Computing for Science
Environment Resource control
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
“Workspaces”
Dynamically provisioned environments Environment control Resource control
Implementations Via leasing hardware platforms: reimaging,
configuration management, dynamic accounts…
Via virtualization: VM deployment
Isolation
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
A Brief History of Nimbus
Research on agreement-based
services
Xen released
First WSRFWorkspace Service
release
EC2 gatewayavailable
Support for EC2 interfaces
2003 20092006
EC2 goes online
First STAR productionrun on EC2
Nimbus Cloudcomes online
Context Brokerrelease
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Nimbus Goals
Allow providers to build clouds Private clouds (privacy, expense considerations) Workspace Service: open source EC2 implementation
Allow users to use cloud computing Do whatever it takes to enable scientists to use IaaS Context Broker: turnkey virtual clusters IaaS Gateway: interoperability
Allow developers to experiment with Nimbus For research or usability/performance
improvements Community extensions and contributions
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
VWSService
The Workspace Service
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
The Workspace Service
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
Poolnode
The workspace service publishesinformation about each workspace
Users can find outinformation about theirworkspace (e.g. what IP
the workspace was bound to)
Users can interact directly with their
workspaces the same way the would with a
physical machine.
VWSService
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Workspace Service: Interfaces and Clients
Web Services based Web Service Resource Framework (WSRF)
WS + state management (WS-Notification) Elastic Computing Cloud (EC2)
Compatible with EC2 clients Supported: ec2-describe-images, ec2-run-instances, ec2-
describe-instances, ec2-terminate-instances, ec2-reboot-instances, ec2-add-keypair, ec2-delete-keypair
Unsupported: availability zones, security groups, elastic IP assignment, REST
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Workspace Service: Security
GSI authentication and authorization PKI-based VOMS, Shibboleth (via GridShib), custom PDPs
Secure access to VMs EC2 key generation or accessed from .ssh
Validating images and image data Extensions from Vienna University of Technology Paper: Descher et al., Retaining Data Control in
Infrastructure Clouds, ARES (the International Dependability Conference), 2009.
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Workspace Service: Networking
Network configuration External: public IPs or private IPs (via VPN) Internal: private network via a local cluster
network Each VM can specify multiple NICs mixing
private and public networks (WSRF only) E.g., cluster worker nodes on a private
network, headnode on both public and private network
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Workspace Components
workspacecontrol
workspaceresourcemanager
workspacepilot
workspaceclient
workspaceservice
EC2
WSRF
OpenNebulaProject
See papers at: http://workspace.globus.org/papers/index.html• “Simple Leases with Workspace Pilot” (EuroPar08) • “Combining Batch Execution and Leasing Using Virtual Machines” (HPDC08),
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Cloud Capabilities
workspacecontrol
workspaceresourcemanager
workspacepilot
workspaceclient
cloudclient
storageservice
workspaceservice
EC2
WSRF
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
The IaaS Gateway
IaaSgateway
EC2potentially other providers
workspacecontrol
workspaceresourcemanager
workspacepilot
workspaceclient
cloudclient
storageservice
workspaceservice
EC2
WSRF
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
User Environments
Cloud Computing Ecosystem
Appliance ProvidersMarketplaces, commercial providers,
Virtual OrganizationsAppliance management software
Deployment Orchestrator
VMM/DataCenter/IaaS User EnvironmentsVMM/DataCenter/IaaS
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
MPIMPI
Turnkey Virtual Clusters
Turnkey, tightly-coupled cluster Shared trust/security context Shared configuration/context information
IP1IP1 HK1HK1
IP1IP1
IP2IP2
IP3IP3
HK1HK1
HK2HK2
HK3HK3
Context Context BrokerBroker
IP2IP2 HK2HK2
IP1IP1
IP2IP2
IP3IP3
HK1HK1
HK2HK2
HK3HK3
IP3IP3 HK3HK3
IP1IP1
IP2IP2
IP3IP3
HK1HK1
HK2HK2
HK3HK3
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Context Broker Goals
Can work with every appliance Appliance schema, can be implemented in
terms of many configuration systems Can work with every cloud provider
Simple and minimal conditions on generic context delivery
Can work across multiple cloud providers, in a distributed environment
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Context Broker Status
Releases In alpha since 08/07, first release 06/08, update
01/09 Used to contextualize cluster composed of 100s of
virtual nodes for multiple production apps Contextualized images on workspace marketplace Working with rPath to make contextualization
easier for the user Discussing OVF extensions
Paper: Keahey&Freeman, Contextualization: Providing One-Click Virtual Clusters, eScience 2008
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
End of Nimbus Tour
workspacecontrol
workspaceresourcemanager
workspacepilot
workspaceservice
workspaceclient
cloudclient
IaaSgateway
cont
ext
brok
er
contextclient
EC2potentially other providers
storageservice
EC2
WSRF
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Science Clouds
Goals Enable scientific projects to experiment with IaaS clouds Evolve software in response to the needs of scientific projects A laboratory for exploration of cloud interoperability issues
Participants University of Chicago (since 03/08, 16 nodes), University of Florida
(05/08, 16-32 nodes, access via VPN), Masaryk University, Brno, Czech Republic (08/08), Wispy @ Purdue (09/08)
In progress: Grid5K, Vrije, others Using EC2 for large runs
Simple governance model, access given to any scientific project http://workspace.globus.org/clouds
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Who Runs on Nimbus at UC?
HadoopAliEnGT-scalabilitySTARMontage workflowsGridFTP testingworkspace-teamTestingOSGgeofestbioinformaticsOther
100+ DNs
projects ranging across Science, CS, education, build&test…
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
STAR
STAR: a nuclear physics experiment studies fundamental properties of nuclear matter
Computations require complex and consistently configured environments
Requirements A virtual OSG STAR cluster: OSG headnode (gridmapfiles,
host certificates, NFS, Torque), worker nodes: SL4 + STAR From Science Clouds to EC2 runs One-click virtual cluster deployment: Context Broker
Producing just-in-time results for Quark Matter conference: http://www.isgtw.org/?pid=1001735
Work by Jerome Lauret, Doug Olson, Leve Hajdu, Lidia Didenko at BNL
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Alice HEP Experiment at CERN
Collaboration with CERNVM project HPCwire article
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Sky Computing
U of FloridaU of Chicago
ViNErouter
ViNErouter
ViNErouter
Purdue
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Sky Computing
Papers: “Sky Computing”, by K. Keahey, A. Matsunaga, M. Tsugawa, J.
Fortes. Submitted to IEEE Internet Computing. “CloudBLAST: Combining MapReduce and Virtualization on
Distributed Resources for Bioinformatics Applications” by A. Matsunaga, M. Tsugawa and J. Fortes. eScience 2008.
U of FloridaU of Chicago
Purdue
Hadoop cloud
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
IaaS Clouds vs Grids Grid computing
Assumption: site retains control over resources Remote interfaces to local site mechanisms Tradeoff: difficult to provide the right environments and control but
easy to deploy Cloud computing
Assumption: a user gets a “lease” on a remote resource that it gets to control
Enabled by virtauliaztion (Xen) Tradeoff: eanbles a larger class of applications but hard to deploy Raises issues: e.g., site licenses? Configuration support?
Towards “sky computing” I can now trust a remote resource: I configured it myself Cloud computing + virtual networks Local distributed environment
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Friends and Family
Committers: Kate Keahey & Tim Freeman (ANL/UC), Ian Gable (UVIC)
A lot of help from the community, see: http://workspace.globus.org/people.html
Collaborations: Cumulus: S3 implementation (Globus team) EBS: IU project Appliance management: rPath, Bcfg2 project, CohesiveFT Virtual network overlays: University of Florida Security (research): Vienna University of Technology
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Open Source IaaS Implementations
OpenNebula Open source datacenter implementation University of Madrid, I. Llorente & team, 03/2008
Eucalyptus Open source implementation of EC2 UCSB, R. Wolski & team, 06/2008
Cloud-enabled Nimrod-G Open source implementation of EC2 Monash University, MeSsAGE Lab, 01/2009
Industry efforts openQRM, Enomalism
04/09/23 The Nimbus Toolkit: http//workspace.globus.org
Parting Thoughts
Science-driven cloud computing Importance of open source
Drive requirements into the infrastructure, customize Drive the development of standards
Cloud computing for the user Combine with what we have (grid computing) Explore new potential
Future directions Creating the ecosystem, working out the issues, e.g.
licensing, appliance support Interoperability and standards Service Levels
top related