networking is not free: lessons in network design
DESCRIPTION
An in-depth critique of the existing OpenStack networking approach, with a focus on how the Nova network controller is more of a hindrance than a help. Discusses the gap in Quantum's functionality required to close the gap, and alternative solutions. How can we make networking in OpenStack robust, high performance, and fault tolerant? What do typical large scale networks look like and what lessons can we learn from them? Is there an approach to networking we can take that is the same with a handful of servers as it is with hundreds of racks?TRANSCRIPT
![Page 1: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/1.jpg)
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution** All unlicensed or borrowed works retain their original licenses
Dan Sneddon
Member Technical Staff
Twitter: @dxs
Download: http://engineering.cloudscaling.com/portland13
Networking is NOT Free: Lessons In Network DesignNetworking is NOT Free: Lessons In Network Design
![Page 2: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/2.jpg)
Presenter Bio
2
Dan Sneddon
Member Technical Staff
Twitter: @dxs
• 20 years of network engineering and systems design
• Lead Global Network Engineer for Apple
• Network Security Architect for SLAC National Laboratory
• IT Architect for division of Schneider Electric
• Financial sector networking (banks and trading floors)
• Major startups, including Twitter
![Page 3: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/3.jpg)
Our Journey Today
3
1. Datacenter Networking: Historical Perspective
2. Rise and Fall Of The VLANs
3. Networking At Cloud Scale
4. OpenStack Networking Models
5. Room For Improvement In OpenStack Networking
![Page 4: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/4.jpg)
Datacenter Networks: Historical Perspective
![Page 5: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/5.jpg)
Datacenter Networking Timeline
5
2000's1980's
2010+1990's
•Client/Server
•10 Mb Ethernet
•Token Ring
•Serial Cables
•100 Mb
•Switched Ethernet
•Bonded Interfaces
•Spanning-Tree
•1 Gb+ Servers
•10 Gb Uplinks
•VLANs
•Virtual Machines
•10 Gb+ Servers
•40/100 Gb Uplinks
•Virtual Networks
•SDN
![Page 6: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/6.jpg)
6
1980’s: Shared Media and Serial
Token Ring
10 M Hub
10M Hub
Serial LinkUser
![Page 7: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/7.jpg)
1990‘s: 100 Megabits Switched!
7
User
User
User
Database
Switch
![Page 8: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/8.jpg)
2000’s: Rise Of the Gigabit VLANs!
8
VLAN 10
VLAN 20
VLAN 30
Etc...
Database
Server VLAN
Administration
Accounting
Everyone Else
![Page 9: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/9.jpg)
2010’s: Everything Gets Simple!
9
User
![Page 10: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/10.jpg)
Rise And FallOf the VLANS
![Page 11: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/11.jpg)
Datacenter VLAN Segregation
11
VLAN 10 VLAN 20 VLAN 30
Layer 2/3 Boundary
![Page 12: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/12.jpg)
VLAN Physical Separation
12
![Page 13: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/13.jpg)
VLAN Pros and Cons
13
• Provide a level of isolation
• Reduction in size of broadcast domain
• Manageable, up to a certain size (especially with VTP, etc)
Pros:
• Each VLAN can only reach other VLANs through routers
• Spanning-tree (when it breaks, everything breaks)
• 4096 VLAN limit--assigning in blocks uses this up faster
Cons:
![Page 14: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/14.jpg)
VLANs Only Scale So Far
• In the late 2000’s, high-density (1U) servers become standard
• There is no way to make spanned VLANs work for many thousands of servers
• A new model takes over: small layer 2 domains with layer 3 routing
Death Of the VLANs
14
![Page 15: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/15.jpg)
Breaking Through The Scale Barrier
15
VLANs Only Scale So Far
VLAN Locally, Route Globally
![Page 16: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/16.jpg)
Hierarchical Internetworking Model
16
Core
Distribution
Access
Hosts
Scale-Out Networking
![Page 17: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/17.jpg)
Networking At the Scale of Cloud
![Page 18: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/18.jpg)
Two Cloud Infrastructure Models
18
Legacy Apps
EnterpriseVirtualization
1
NewDynamic Apps
ElasticInfrastructure
2
![Page 19: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/19.jpg)
Elastic Cloud vs.Enterprise Virtualization
19
Enterprise Virtualization Elastic Cloud
Applications Traditional & Legacy Dynamic
Scaling Architecture Managed Silos Horizontal
Technology Stack Heavy & Proprietary Distributed & Open
Price/Performance Low High (4-7x better)
Failure Domains Large Small
Provisioning Slower & Manual Faster & 100% API
Best For: Server consolidation and lower datacenter mgmt costs
On-demand, scale-out infrastructure for new apps
![Page 20: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/20.jpg)
Classic OpenStack Networking, With That Old-Timey Feel
• Flat/Flat DHCP only support a single VLAN for everything
• VlanManager is the most feature-rich for multi-tenant
• VlanManager requires trunking all VLANs down to each host
• In a public cloud, max of 4096 VLANs limits tenants
Nova-Network
20
Flat Flat DHCP VlanManager FlatDHCP Multi-host HA4 Modes:
![Page 21: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/21.jpg)
OCS Nova-Networking L3 Plugin
21
Cloudscaling Exclusive Solution
• Layer 3 networking for VMs, with DHCP and NAT service
• Each VM is on its own Linux bridge, no shared layer 2
• Quantum not required
• DHCP service is local to each compute host
• AWS-like: floating IPs, elastic netblocks, and now VPC
![Page 22: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/22.jpg)
Brokerless Messaging With ZeroMQ
22
Avoiding RabbitMQ’s Single Point Of FailureNova-Compute
Nova-Scheduler Nova-API
RabbitMQBroker
RabbitMQ(Brokered)
Single Point Of Failure
Nova-Compute
Nova-Scheduler Nova-API
vs. ZeroMQ(Peer To Peer)
![Page 23: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/23.jpg)
OpenStack Networking
![Page 24: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/24.jpg)
APIs For All Your Networking Things
• “Quantum” is now known as “OpenStack Networking”
• Pluggable architecture, with APIs for all network functionality
• Basic L3 plugin (finally!), but designed for L3 on flat L2 network
• nova-network process still performs some very basic functions
• Some plugins are more complete/stable than others
OpenStack Networking
24
![Page 25: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/25.jpg)
OpenStack Networking
25
Quantum DB
Quantum API Service
QuantumAgent(s)
RPC
SQL
Varies
REST
Horizon
REST over HTTP(S)
Nova(Quantum
Plugin)REST
Keystone
Ceilometer
REST
Notifi-cations
compute node
Hypervisor
Virtual Network Plugin
Provider Network Plugin
DHCP Agent
SDN Solution
Physical Hardware
Varies
Varies
OPENSTACKNETWORKSERVICE
![Page 26: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/26.jpg)
OpenStack Networking Modes
26
• VLAN networks are supported using provider network plugins
• Layer 3 plugin
• GRE tunnel support using virtual network plugins
• May be used with Linux Namespaces to isolate tenants from one another within a hypervisor
• Many commercial vendor plugins
![Page 27: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/27.jpg)
Quantum Compatibility
27
Lots Of Choices For Virtual Network/SDN Providers
•Open vSwitch. http://www.openvswitch.org/openstack/documentation
•Nicira NVP. quantum/plugins/nicira/nicira_nvp_plugin/README and http://www.nicira.com/support.
•Midokura. http://www.midokura.com/midonet/openstack/
•BigSwitch. http://www.bigswitch.com/sites/default/files/sdn_resources/openstack_aag.pdf
•Cisco. quantum/plugins/cisco/README and http://wiki.openstack.org/cisco-quantum
•Linux Bridge. quantum/plugins/linuxbridge/README and http://wiki.openstack.org/Quantum-Linux-Bridge-Plugin
•Ryu. quantum/plugins/ryu/README and http://www.osrg.net/ryu/using_with_openstack.html
•NEC OpenFlow. http://wiki.openstack.org/Quantum-NEC-OpenFlow-Plugin
![Page 28: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/28.jpg)
Room For Improvement
![Page 29: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/29.jpg)
29
Default Layer 3 Design
VLANs
OpenStack Networking Won’t Magically Configure Routing
* Diagram taken from OpenStack Networking official documentation
![Page 30: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/30.jpg)
Gaps In Functionality
30
• VLAN networks are still problematic, Quantum doesn’t fix that
• Layer 3 network plugin still gets deployed on shared layer 2
• Dynamic routing protocols are not supported by L3 plugin
• Overlay networks are great, unless something goes wrong--GRE tunnels hard to troubleshoot, we need tooling, diagnostics
• Load-balancer-, firewall-, and VPN-as-a-service still in design phase, may not be production-ready until I or J release
![Page 31: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/31.jpg)
How Can We Make Things Better?
31
• Further work needed on the “metaplugin” that allows more than one plugin simultaneously
• ZeroMQ support (there are known problems with DHCP, etc.)
• Better high-availability, including active-active DHCP
• Better support for custom tenant networks with overlapping IPs
There Are Plenty Of Ways To Contribute
![Page 32: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/32.jpg)
32
Questions
![Page 33: Networking is NOT Free: Lessons in Network Design](https://reader034.vdocuments.mx/reader034/viewer/2022052618/554bba9cb4c90594278b4c63/html5/thumbnails/33.jpg)
CCA - NoDerivs 3.0 Unported License - Usage OK, no modifications, full attribution** All unlicensed or borrowed works retain their original licenses
Networking is NOT Free: Lessons In Network Design
Dan Sneddon
Member Technical Staff
Twitter: @dxs
Download: http://engineering.cloudscaling.com/portland13