gec3 oct. 28-30 2008 experiment workflow and services working group system engineering report vicraj...
TRANSCRIPT
GEC3 Oct. 28-30 2008 www.geni.net 1
Experiment Workflow and Services Working Group
System Engineering Report
Vicraj (Vic) Thomas, Ph.D.Working Group System Engineer
groups.geni.netGENI working group wiki
GEC3 Oct. 28-30 2008
We Are Here…
www.geni.net 2
The Experiment Services and Workflow WG focuses on experimenter-users needs for planning, scheduling, running, debugging, analyzing and archiving experiments.
The Experiment Services and Workflow WG focuses on experimenter-users needs for planning, scheduling, running, debugging, analyzing and archiving experiments.
GEC3 Oct. 28-30 2008
Lifecycle of a GENI Experiment
• User view of an experiment• Tools and services supporting this lifecycle map to
different subareas identified by Jeff Chase– Slice control
• Construct, deploy, monitor and adapt slices– Information plane
• Instrumentation: Specification, collection, processing– Experiment building blocks
• Code artifacts: Specification, qualification, sharing/reuse, deployment, archival, etc.
• Objective: Think about how your ideas and projects relate to workflow presented here– How they support this workflow– Identify areas of disagreement
www.geni.net 3
GEC3 Oct. 28-30 2008
Lifecycle of a GENI Experiment: Top-Level View
www.geni.net 4
ExperimentDesign
S/w and H/wDevelopment (Lab)
Experiment Planning
Experiment Execution
Experiment Sunsetting
Experiment Deployment
GENI Tools and Services support these phases of the lifecycle
Outside the scope of GENI
User Registration
GEC3 Oct. 28-30 2008
GENI Relevant Lifecycle Phases
www.geni.net 5
Establish Credentials
Experiment Control
Archive Experiment
Establish Credentials
Resource Discovery
Tool Discovery
Develop GENI Version
Lease Resources
Install and Configure Software
Data Collection/Analysis
Transition to Product
Experiment Planning
Experiment Deployment
Experiment Execution
Experiment Sunsetting
Verify Deployment
GEC3 Oct. 28-30 2008
User Registration
• Needed: Admin tools to register users and grant them appropriate authorizations– Administrator uses real-
world documents to authenticate user
– User is registered with a Principal Registry/Identity Provider
– User is granted appropriate authorizations
• Browse resources• Lease resources and run
experiments
www.geni.net 6
GENI Clearinghouse
ComponentRegistry
Principal Registry
●■Ticket: Store Query
●■Principal: Register Authen Query
Slice Registry
TicketLog
SoftwareReposit
●■Slice: Register Cred Issue or Tkt Broker Query
●■Soft Mod: Store Get Query
●■Component: Register Query
Admin and AccountTools
GEC3 Oct. 28-30 2008
Experiment Planning
www.geni.net 7
Establish Credentials
Experiment Control
Archive Experiment
Establish Credentials
Resource Discovery
Tool Discovery
Develop GENI Version
Lease Resources
Install and Configure Software
Data Collection/Analysis
Transition to Product
Experiment Planning
Experiment Deployment
Experiment Execution
Experiment Sunsetting
Verify Deployment
GEC3 Oct. 28-30 2008
Establishing Credentials
• User establishes identity with Clearinghouse– Assumption: Already registered with an Identity
Provider/Principal Registry
www.geni.net 8
GEC3 Oct. 28-30 2008
Resource Discovery
• Multiple ways of finding resources– Automatically from an experiment specification
• Needed: A tool that can interpret an experiment specification and query appropriate clearinghouses for resources
– Programmatically• Needed: An API on Clearinghouses to query for resources
– Browse and select• Needed: A graphical tool for browsing resources
– May lease resources during discovery (for immediate or future use)
• Researcher credentials must allow leasing
• Assumption: All components and aggregates are registered with one or more Clearinghouses
www.geni.net 9
GEC3 Oct. 28-30 2008
Tool Discovery
• Determine tools needed to program GENI components discovered by researcher– Need: Component Rspec to provide links to
information on using/programming component– Need: Component Rspec to provide links to
information on resource and other constraints imposed by component
www.geni.net 10
GEC3 Oct. 28-30 2008
Develop GENI Version
• Version of experiment that will run on GENI components– Developed and tested in the lab
• Based on component information obtained during resource and tool discovery
• Develop instrumentation plan and implement• GENI Tools/Services: None needed
www.geni.net 11
GEC3 Oct. 28-30 2008
GENI Relevant Lifecycle Phases
www.geni.net 12
Establish Credentials
Experiment Control
Archive Experiment
Establish Credentials
Resource Discovery
Tool Discovery
Develop GENI Version
Lease Resources
Install and Configure Software
Data Collection/Analysis
Transition to Product
Experiment Planning
Experiment Deployment
Experiment Execution
Experiment Sunsetting
Verify Deployment
GEC3 Oct. 28-30 2008
Establish Deployment Credentials
• Assumption: Researcher has rights to lease resources– Credentials to lease resources and use them may be
different from those for browsing resources• Needed: Researcher’s Principal Registry/Identity
Provider to be trusted by Clearinghouses
www.geni.net 13
GEC3 Oct. 28-30 2008
Lease Resources
• Similar to browsing for resources– Automatically from an
experiment specification• Needed: Tool to interpret
experiment specification and lease resources at appropriate Clearinghouses
– Programmatically• Needed: API on Clearinghouses to
lease resources– Browse and lease
• Needed: Graphical tool to lease resources
• Researcher specifies duration of lease and start time
www.geni.net 14
GENI Clearinghouse
ComponentRegistry
Principal Registry
●■Ticket: Store Query
●■Principal: Register Authen Query
Slice Registry
TicketLog
SoftwareReposit
●■Slice: Register Cred Issue or Tkt Broker Query
●■Soft Mod: Store Get Query
●■Component: Register Query
GENI Aggr A
Host A1
Host Ax
● ■Aggr Mgr with Ops Portal
Admin Oper
GENI Comp B
Comp B
● ■Comp Mgr with Ops Portal
Admin Oper
Research Org A
● Researcher
● Slice Admin
● PI
LocalPrincipal Registry
Leasing Interface
Details of Control Plane operations(slice management, tickets, etc) are hidden from the casual user
GEC3 Oct. 28-30 2008
Install and Configure Software
• Download software into components and install them– Experimenter must have valid lease for resource
• Configure components such as firewalls and Internet gateways
• Needed: Tool to manage downloads, track configurations, etc.
• Needed: Mechanisms for composing experiments
• Experimental hardware: Specific to type of hardware and where it goes– Post Spiral 1
www.geni.net 15
GEC3 Oct. 28-30 2008
Validate Deployment
• Needed: Tools to ensure slice is properly configured– Required resources have been obtained– Components connected and communicating
www.geni.net 16
GEC3 Oct. 28-30 2008
GENI Relevant Lifecycle Phases
www.geni.net 17
Establish Credentials
Experiment Control
Archive Experiment
Establish Credentials
Resource Discovery
Tool Discovery
Develop GENI Version
Lease Resources
Install Hardware/Software
Data Collection/Analysis
Transition to Product
Experiment Planning
Experiment Deployment
Experiment Execution
Experiment Sunsetting
Verify Deployment
GEC3 Oct. 28-30 2008
Experiment Control
• Needed: Services and tools to support– Start/pause/resume/stop experiments
• Granularity: Slice? Sliver? Experiment relevant groupings? All of the above?
– Grow/shrink slice: Add/remove resources
– Control flow of traffic from/to Internet
– Connect to other experiments (experiment composition)
– Distributed synchronization for multi-phased experiments
– Debugging and checkpointing• Resources being used by experiment• Experiment state
– Failure management• Handling failures of components and links:
www.geni.net 18
GEC3 Oct. 28-30 2008
Data Collection and Analysis
• Needed: Non-intrusive instrumentation– Data collection and storage
• Needed: Tools for monitoring and visualizing resources, flows, etc. in real-time while experiment is running
www.geni.net 19
GEC3 Oct. 28-30 2008
GENI Relevant Lifecycle Phases
www.geni.net 20
Establish Credentials
Experiment Control
Archive Experiment
Establish Credentials
Resource Discovery
Tool Discovery
Develop GENI Version
Lease Resources
Install Hardware/Software
Data Collection/Analysis
Transition to Product
Experiment Planning
Experiment Deployment
Experiment Execution
Experiment Sunsetting
Verify Deployment
GEC3 Oct. 28-30 2008
Experiment Archival
• Make experiment available to the research community to replicate, extend or incorporate into another experiment
• Needed: Checklist of tasks to be completed before experiment is archived– Documentation of experiment objectives, code,
resources used; keywords; experiment description in a formal specification language; published reports; etc.
• Needed: Tools to search archives• Needed: Embedding experiment specifications
www.geni.net 21
GEC3 Oct. 28-30 2008
Transition Technology to Product
• Outside scope of GENI tools and services?
www.geni.net 22
GEC3 Oct. 28-30 2008
Working Group Deliverables and GPO Support
www.geni.net 23
GEC3 Oct. 28-30 2008
WG Deliverables
• GENI Experiment Lifecycle• Spiral 1 Capabilities-based Use Case
– Control plane oriented
• Experiment Support Subsystem Technical Description
• Experiment Support Tools and Services Technology Survey
www.geni.net 24
GEC3 Oct. 28-30 2008
My Role in this Working Group
• Frame issues from top-down through document development– Collect & frame issues
– Collect input from group & revise
• Synthesize input from bottom-up– Test for & document consensus (or lack of it)
– Take & distribute notes
– Maintain wiki
• Track hard problems and open issues– Solicit help with developing cogent descriptions of problems
– Identify researchers and projects that can address these problems
www.geni.net 27
GEC3 Oct. 28-30 2008
GPO System Engineering and Working Groups
• SEs draft documents• Internal GPO review• Post publically
– On wiki initially, repository up shortly
• Discuss doc on working group list– Possible one-on-one followups– Please send text
• Revision is posted– Repeat process
www.geni.net 28
GEC3 Oct. 28-30 2008GEC3 www.geni.net 29
GENI Spiral 1 Integration: 5 Control Framework Clusters
1578 Overlay Hosting Nodes1578 Overlay
Hosting Nodes
1579 ProtoGENI
1579 ProtoGENI
1595 GpENI1595 GpENI
1646 CMU Testbeds
1646 CMU Testbeds
1609 DETERTrial
Integration
1609 DETERTrial
Integration
1642 Instrumentation
Tools
1642 Instrumentation
Tools
1613 Enterprise
GENI
1613 Enterprise
GENI
1600 PlanetLab
1600 PlanetLab
1658 Mid-Atlantic Crossroads
1658 Mid-Atlantic Crossroads
1602 Sensor/Actuat
or Network
1602 Sensor/Actuat
or Network
1582 ORCA/BEN
1582 ORCA/BEN
1599 Vehicular Mobile
Network
1599 Vehicular Mobile
Network
1660 ORBIT Framework
1660 ORBIT Framework
1601 Virtual Tunnels
1601 Virtual Tunnels
1657WIMAX
1657WIMAX
1663 Digital Object
Registry
1663 Digital Object
Registry
1633 Kansei Sensor Network
1633 Kansei Sensor Network
1628 Measurement
System
1628 Measurement
System
1604 GENI Meta
Operations
1604 GENI Meta
Operations
1643 Programmable
Edge Node
1643 Programmable
Edge Node
1621 GUSH Tools
1621 GUSH Tools
1622 Provisioning
Service
1622 Provisioning
Service
1650 Regional
Opt-In
1650 Regional
Opt-In
1632 Security Architecture
1632 Security Architecture
1645 Million- Node GENI1645 Million- Node GENI
1610 GENI at 4-
Year Colleges
1610 GENI at 4-
Year Colleges1653 Data Plane
Measurements
1653 Data Plane
Measurements
Cluster A Cluster ECluster D Cluster CCluster B
Key:
Projects with active Spiral 1clearinghouse interfaces
STUDY ALL PICK ONE
1631 Embedded Real-time
Measurements
1631 Embedded Real-time
Measurements
1619 Optical Access
Networks
1619 Optical Access
Networks
Column labels showcommon control framework
Highlighted projects are relevant to this wg. Is this the right set?Highlighted projects are relevant to this wg. Is this the right set?
GEC3 Oct. 28-30 2008
Follow-up
• Notes, slides, actions, etc will be sent to the working group mail list and posted on the wiki page: http://www.geni.net/wg/services-wg.html
• Please give Jeff/me copies of your slides immediately after this session
• Please sign Non-Exclusive Copyright License
www.geni.net 30