ocean observatories initiative common execution environment kate keahey ooi cyberinfrastructure life...
TRANSCRIPT
Ocean Observatories InitiativeOcean Observatories Initiative
Common Execution Environment
Kate Keahey
OOI Cyberinfrastructure Life Cycle ObjectivesMilestone Review, Release 1
San Diego, CAFebruary 23-25, 2010
OOI CI LCO Review, Feb 2010
2
Overview
• Use Cases
• Architecture Overview
• CEI Components in R1
• CEI Elements– Elastic Provisioning Unit– Infrastructure as a Service– Deployable Type management
Use Case Overview
OOI CI LCO Review, Feb 2010
3
- Highly Available Services- Rapidly provision resources- Scale to demand
Architecture Overview
OOI CI LCO Review, Feb 2010
4
EPU
EPU Worker(Operational Unit)
EPU Worker(Operational Unit)
EPU Worker(Operational Unit)
EPU Worker(Operational Unit)
EPU Worker(Operational Unit)
EPU Worker(Operational Unit)
HA Service(OOI Application)
VM(Deployable Unit) VM
(Deployable Unit) VM(Deployable Unit)
Application Software
(Deployable Type)
Architecture Overview
OOI CI LCO Review, Feb 2010
5
Taskable Resources
OOI CI LCO Review, Feb 2010
6
ProcessProcessOperational Unit Operational Unit HA ServiceHA Service
Simple or complex deployed appliance, e.g., a deployed VM instance, a virtual cluster, platforms for workflow execution
e.g., an executing program or a long-running service
e.g., a core or application service
Deployable Type Deployable Type
Deployable Unit Deployable Unit
e.g., a description of an environment in rBuilder
e.g., a VM image
Taskable Resources
Start/StopStart/Stop MonitorableMonitorable ControllableControllable
Architecture Overview
OOI CI LCO Review, Feb 2010
7
Architecture Overview
OOI CI LCO Review, Feb 2010
8
OOI CI LCO Review, Feb 2010
9
CEI Service Interactions
Planner
Controller(“Executive”)
ProvisionerFault Monitoring
andCompensation
Registry
Resource(Taskable)
2970-00001 OV2 CEI
CEI Service Interaction
OOI CI LCO Review, Feb 2010
10
Planner
ControllerProvisioner
Monitoring Information
Deployable Type
Registry
Context Broker
IaaS Adapter IaaS
ExchangeExchange
VM infoVM info
Proc InfoProc Info
Other InfoOther Info
Elastic Processing Unit• Elastic Processing Unit (EPU): High-
Availability Services with policy based scaling to demand
OOI CI LCO Review, Feb 2010
11
EPU Controller
Register HA service
EPU WorkerN1
EPU WorkerN1
Msg to N
EPU WorkerN2
EPU WorkerN2
EPU WorkerNX
EPU WorkerNX
…
Elastic Processing Unit
OOI CI LCO Review, Feb 2010
12
Planner
ControllerProvisioner
Monitoring Information
Deployable Type
Registry
Context Broker
IaaS Adapter IaaS
ExchangeExchange
VM infoVM info
Proc InfoProc Info
Other InfoOther Info
Will this whole thing work (even in a small way)?
Will this whole thing work (even in a small way)?
EPU: Prototype 1• EPU design and prototype:
– Policy-configurable planner/scheduler– Scaling policy: size of work queue– All components integrated with Magnet/AMQP
OOI CI LCO Review, Feb 2010
13
EPU: Prototype 2• Updates technology choices
– Uses Redis to distribute work to VM instances and as Information Aggregator
• New policy– load
• Demonstration later
OOI CI LCO Review, Feb 2010
14
EPU: Technology Choices• IaaS Adapter: still evaluating
– Deltacloud, jcloud, libcloud, dasein, fog – Standards: OCCI-WG
• Nimbus Context Broker
• Redis for Information storage
OOI CI LCO Review, Feb 2010
15
EPU: Elaboration Iterations• Atomic HA for EPU controller
• Detailed design on interaction with COI
• Scalability, scalability, scalability
• Finalize technology choices
• Prototype end-to-end EPU
• Stress testing and harness
• Investigate storage/compute/network
• Refine implementation
OOI CI LCO Review, Feb 2010
16
Infrastructure-as-a-Service
OOI CI LCO Review, Feb 2010
17
Planner
ControllerProvisioner
Monitoring Information
Deployable Type
Registry
Context Broker
IaaS Adapter IaaS
ExchangeExchange
VM infoVM info
Proc InfoProc Info
Other InfoOther Info
Will it work with COI?Will it work with COI?
Where are we going to get IaaS resources?
Where are we going to get IaaS resources?
Can we adapt existing resources?
Can we adapt existing resources?
Infrastructure-as-a-Service• Targeted deployment platforms
– Commercial (AWS, rackspace)– Scientific (FutureGrid, Magellan, …)
• IaaS for science– The Nimbus Workspace Service– Evaluate Ganeti/SGE/Condor
• Nimbus– Make implementation technology-independent – Make implementation WSRF-independent – Provide user management functions
OOI CI LCO Review, Feb 2010
18
Deployable Types
OOI CI LCO Review, Feb 2010
19
Planner
ControllerProvisioner
Monitoring Information
Deployable Type
Registry
Context Broker
IaaS Adapter IaaS
ExchangeExchange
VM infoVM info
Proc InfoProc Info
Other InfoOther Info
complexity?complexity?
Will it work with COI?
Will it work with COI? scalability?scalability?
Deployable Types• Complexity
– Document requirements and use cases• Web App, Hadoop, Job Queue, NFS, Torque
– Design overall approach– Preliminary roundup and evaluation of
relevant technologies • Puppet, Chef, CohesiveFT, rBuilder• Working choices: rBuilder and Chef
– Prototype integration of Chef with the Nimbus Context Broker
OOI CI LCO Review, Feb 2010
20
Deployable Types• Protocol-independence
– Make the Context Broker WSRF-independent– REST interfaces
• Scalability and Sustainability– Prototype scalability harness– Scalability tests
• Contextualizing up to 500 nodes • Challenges in harness design and efficiency
OOI CI LCO Review, Feb 2010
21
DT: Elaboration Iterations• Finalize technology choices for
management of Deployable Types/Units
• Deployable Type Registry
• More comprehensive harness and more contextualization scalability tests
• Understanding performance
OOI CI LCO Review, Feb 2010
22
OOI CI LCO Review, Feb 2010
23
Thanks !