open innovation platforms: network testbeds, cloud ......www. chameleoncloud.org chameleon: a...
TRANSCRIPT
![Page 1: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/1.jpg)
Open Innovation Platforms: Network
Testbeds, Cloud Testbeds and StarLight Software Define Network Exchange(StarLight SDX)
Jim ChenAssociate Director
International Center for Advanced Internet Research(iCAIR)Northwestern University
APAN41-Cloud WGManila, PhilippinesJanuary 26, 2016
![Page 2: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/2.jpg)
• Creation and Early Implementation of Advanced Networking Technologies - The Next Generation Internet All Optical Networks, Terascale Networks, Networks for Petascale Science
• Advanced Applications, Middleware, Large-Scale Infrastructure, NG Optical Networks and Testbeds, Public Policy Studies and Forums Related to NG Networks
• Three Major Areas of Activity: a) Basic Research b) Design and Implementation of Prototypes c) Operations of Specialized Communication Facilities (e.g., StarLight)
Accelerating Leading Edge Innovation
and Enhanced Global Communications
through Advanced Internet Technologies,
in Partnership with the Global Community
Introduction to iCAIR:
![Page 3: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/3.jpg)
Open Innovation Platforms
• GENI: Global Environment for Networking Innovations, “a virtual laboratory for exploring future internet technologies and protocols at scale”.
• NSF13602 NSFCloud: “enable the academic research community to develop and experiment with novel cloud and cloud-like architectures that can support a diversity of innovative applications”.
• NSF14554 IRNC RXP: “platforms of network innovation supporting the research and education needs of global scientific collaborations”.
![Page 4: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/4.jpg)
![Page 5: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/5.jpg)
![Page 6: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/6.jpg)
Multiple HPC Cloud Computing Testbeds
Specifically Designed for Science Research
At Scale Experimentation
Integerated With High Performance Networks
=> Open Commons Consortium
![Page 7: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/7.jpg)
![Page 8: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/8.jpg)
Will Be Contiguous To
the StarLight SDX
![Page 9: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/9.jpg)
Global Research Platform
• Currently – A Concept
• A Specialized Globally Distributed Platform For Science Discovery and Innovation
• Based On State-Of-the-Art-Clouds
• Interconnected With Computational Grids, Supercomputing Centers, Specialized Instruments, et al
• Also, Based On World-Wide 100 Gbps Networks
• Leveraging Advanced Architectural Concepts, e.g., SDN/SDX/SDI – Science DMZs
• Ref: Demonstrations @ SC15, Austin Texas November 2015
![Page 10: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/10.jpg)
![Page 11: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/11.jpg)
www. chameleoncloud.org
JANUARY 26 , 2016 11
CHAMELEON: A LARGE-SCALE, RECONFIGURABLE EXPERIMENTAL ENVIRONMENT FOR CLOUD RESEARCHPrincipal Investigator: Kate Keahey
Co-PIs: J. Mambretti, D.K. Panda, P. Rad, D. Stanzione
Presented by Jim Chen, associate Director,
International Center for Advanced Internet Research, Northwestern University
APAN 41
Manila Philippines
JANUARY 25-29, 2016
![Page 12: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/12.jpg)
www. chameleoncloud.org
CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT
Large-scale: “Big Data, Big Compute, Big Instrument research” ~650 nodes (~14,500 cores), 5 PB disk over two sites, 2 sites
connected with 100G network
Reconfigurable: “As close as possible to having it in your lab” From bare metal reconfiguration to clouds
Support for repeatable and reproducible experiments
Connected: “One stop shopping for experimental needs” Workload and Trace Archive
Partnerships with production clouds: CERN, OSDC, Rackspace, Google, and others
Partnerships with users
Complementary: “Can’t do everything ourselves” Complementing GENI, Grid’5000, and other experimental testbeds
![Page 13: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/13.jpg)
www. chameleoncloud.org
CHAMELEON HARDWARE
SCUs connect to core and fully connected to each other
Heterogeneous Cloud Units
Alternate Processors and Networks
SwitchStandard
Cloud Unit42 compute
4 storage
x10
Chicago
To UTSA, GENI, Future Partners
AustinChameleon Core Network
100Gbps uplink public network(each site)
Core Services3.6 PB Central File Systems, Front End and Data Movers
Core ServicesFront End and Data
Mover Nodes 504 x86 Compute Servers48 Dist. Storage Servers102 Heterogeneous Servers16 Mgt and Storage Nodes
SwitchStandard
Cloud Unit42 compute
4 storage
x2
![Page 14: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/14.jpg)
www. chameleoncloud.org
CAPABILITIES AND SUPPORTED RESEARCH
Virtualization technology (e.g., SR-IOV, accelerators), systems, networking, infrastructure-level resource management, etc.
Repeatable experiments in new models, algorithms, platforms, auto-scaling, high-availability, cloud federation, etc.
Development of new models, algorithms, platforms, auto-scaling HA, etc., innovative application and educational uses
Isolated partition, bare metal reconfiguration: OpenStack and Grid’5000
Isolated partition, Chameleon Appliances
Persistent, reliable, shared clouds
![Page 15: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/15.jpg)
www. chameleoncloud.org
SOFTWARE: CORE CAPABILITIES
Chameleon Appliance Catalog
A library of generic, special-purpose, and educational environments
Persistent Clouds
(OpenStack)
Discovery, Provisioning, Configuration, and Monitoring
Testbed representation and discovery (Grid’5000)
Nova/Blazar, Ironic, Neutron, Ceilometer
(OpenStack, Rackspace OnMetal)
Persistent Cloud User Clouds
![Page 16: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/16.jpg)
www. chameleoncloud.org
EXPERIMENTAL WORKFLOW
discover
resources
provision
resources
configure
and interactmonitor
analyze, discuss,
and share
design the
experiment
![Page 17: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/17.jpg)
www. chameleoncloud.org
SELECTING AND VERIFYING RESOURCES
Complete, fine-grained and up-to-date representation Machine parsable, enables match makingVersioned
“What was the drive on the nodes I used 6 months ago?”
Dynamically VerifiableDoes reality correspond to description? (e.g., failures)
Grid’5000 Registry Automated resource description, automated export to RM
G5K-checksRun at boot, acquire information, compare with resource
catalog description
![Page 18: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/18.jpg)
www. chameleoncloud.org
PROVISIONING RESOURCES
Resource leases
Allocating a range of resourcesDifferent node types, switches, etc.
Multiple environments in one lease
Advance reservations (AR) Sharing resources across time
Eventually: match making, Gantt chart displays
OpenStack Nova/BlazarExtensions to support working with more resources, match
making, and displays
![Page 19: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/19.jpg)
www. chameleoncloud.org
CONFIGURE AND INTERACT
Map multiple appliances to a leaseAllow deep reconfiguration (incl. BIOS)SnapshottingEfficient appliance deploymentHandle complex appliances
Virtual clusters, cloud installations, etc.
Interact: reboot, power on/off, access to consoleShape experimental conditions
OpenStack Ironic, Glance, and meta-data servers
![Page 20: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/20.jpg)
www. chameleoncloud.org
MONITORING
Enables users to understand what happens during the experiment
Types of monitoringUser resource monitoringInfrastructure monitoring (e.g., PDUs)Custom user metrics
High-resolution metricsEasily export data for specific experiments
OpenStack Ceilometer
![Page 21: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/21.jpg)
www. chameleoncloud.org
OUTREACH AND ENGAGEMENT
Early User ProgramCommitted users, driving and testing new capabilities,
enhanced level of support
Chameleon WorkshopAnnual workshop to inform, share experimental
techniques solutions and platforms, discuss upcoming requirements, and showcase research
Advisory BodiesResearch Steering Committee: advise on capabilities
needed to investigate upcoming research challengesIndustry Advisory Board: provide synergy between industry
and academia
![Page 22: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/22.jpg)
www. chameleoncloud.org
PARTNERSHIP WITH GENI COMMUNITY
Chameleon Will Enable the GENI Virtual Laboratory For Networking and Distributed Systems Research and Education To Extended Significantly With Many New Types of Resources.
This Blending of Resources Will Enable Investigations Of New Types Of Innovative Highly Distributed Environments at Scale.
The Architectural Options For Blending Chemeleon and GENI Are Being Discussed
![Page 23: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/23.jpg)
www. chameleoncloud.org
EXPERIMENTER RESOURCES
Unique Deeply Programmable Blending Of Resources. Extremely Close Integration (Ideally, Seamless) of
Programmable Networks With Programmable CloudsA Richly Resourced Platform For Experimentation
With Exceptional Novel Architecture.Enables the Creation of New, Highly Innovative
Distributed Environments, Including Specialized/Customized vs Generic Environments
Highly Granulated Levels of Experimental Research Control, Measurement, Analytics, Visualization, and Reproducibility.
![Page 24: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/24.jpg)
www. chameleoncloud.org
PROJECT SCHEDULE
Q1 2015: FutureGrid@ChameleonChameleon Technology Preview OpenStack FutureGrid-style cloud43 projects, 81 users, 29 institutions
Summer 2015: New hardware: large-scale homogenous partitions available to Early Users
Fall 2015: Large-scale homogenous partitions and bare metal reconfiguration generally available
2015/2016: Refinements to experiment management capabilities, higher level capabilities
Fall 2016: Heterogeneous hardware available
![Page 25: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/25.jpg)
www. chameleoncloud.org
TEAM
Kate Keahey
Chameleon PI
Science Director,
Software Development
Joe Mambretti
Programmable networks
DK Panda
High-performance
networks
Dan Stanzione
Facilities Director
Paul Rad
Industry Liason
![Page 26: Open Innovation Platforms: Network Testbeds, Cloud ......www. chameleoncloud.org CHAMELEON: A FLEXIBLE AND POWERFUL EXPERIMENTAL INSTRUMENT Large-scale: “ig Data, ig ompute, ig Instrument](https://reader034.vdocuments.mx/reader034/viewer/2022052022/6037a7ce92d8c8063e02967e/html5/thumbnails/26.jpg)
www. chameleoncloud.org
THE TESTBED IS THERE – “JUST” ADD RESEARCH!
Large-scale, responsive experimental testbed
Targeting critical research problems at scale
Reconfigurable environment
Support use cases from bare metal to production clouds
One-stop shopping for experimental needs
Trace and Workload Archive
Engage the community The most important element of any experimental testbed is users and
the research they work on
Come and visit us at www.chameleoncloud.org!