weird things weve seen with openstack neutron
TRANSCRIPT
OpenStack Neutron• So$ware-defined networking component
• Users define their own virtual networks
• Manages IP address assignment
• Floa?ng IP addresses
• Supports many different back-ends - OpenvSwitch, VMware
NSX, Cisco UCS, Midokura....
Architecture, con-nued
• neutron-{server,agent}
• OpenvSwitch
• Linux bridging
• Linux network namespaces
• L2
• L3
Common problems - typical user complaints• VM can't obtain an IP address
• Can't ping / connect to my VM
• Intermi9ent connec:vity
Weirdness #1 - orphaned namespaces• Default (on Ubuntu) is not to delete namespaces at all (!)
• Bug in iproute2 package
• h=ps://bugs.launchpad.net/neutron/+bug/1052535
• Misconfigured sudo rules meant that network namespaces weren't being deleted
• Mismatch between interfaces configured in a namespace and what Neutron expects
Finding out what's supposed to be wherefor netnode in osnet{0..4} ; do echo $netnode for router in $(ssh $netnode 'ip netns list | grep qrouter | cut -d - -f 2-20') ; do neutron router-show $router | grep -i unable donedone
Then delete each invalid namespace and associated OVS port.
• Pro%p: Don't run neutron-ovs-cleanup!
Weirdness #2 - duplicate segmenta4on ID• Customer support ,cket with instances unable to obtain an IP via
DHCP
• Some serious digging required...
Tracing packet flows• tcpdump on compute node and in network namespaces
• Packets not always arriving where you'd expect
• Have to look at OpenFlow rules
DHCP agentneutron dhcp-agent-list-hosting-net 4dc325ed-f141-41d9-8d0a-4f513defacad+--------------------------------------+--------+----------------+-------+| id | host | admin_state_up | alive |+--------------------------------------+--------+----------------+-------+| 1beb99ef-e6f6-4083-8fb6-661f2f61c565 | osnet1 | True | :-) |+--------------------------------------+--------+----------------+-------+
neutron net-show -F provider:segmentation_id 4dc325ed-f141-41d9-8d0a-4f513defacad+--------------------------+-------+| Field | Value |+--------------------------+-------+| provider:segmentation_id | 11 |+--------------------------+-------+
• 11 in hex = 0xb
root@osnet1:~# ovs-ofctl dump-flows br-tun table=2
NXST_FLOW reply (xid=0x4):cookie=0x0, duration=875584.823s, table=2, n_packets=85, n_bytes=10880, idle_age=11560, hard_age=65534, priority=1,tun_id=0x14 actions=mod_vlan_vid:43,resubmit(,10)cookie=0x0, duration=2578615.436s, table=2, n_packets=1345, n_bytes=128202, idle_age=27174, hard_age=65534, priority=1,tun_id=0x10 actions=mod_vlan_vid:2,resubmit(,10)cookie=0x0, duration=2578611.677s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,tun_id=0xd actions=mod_vlan_vid:12,resubmit(,10)cookie=0x0, duration=1806356.959s, table=2, n_packets=5140, n_bytes=364533, idle_age=341, hard_age=65534, priority=1,tun_id=0x21 actions=mod_vlan_vid:35,resubmit(,10)cookie=0x0, duration=2578610.661s, table=2, n_packets=1035919, n_bytes=180430025, idle_age=65534, hard_age=65534, priority=1,tun_id=0x11 actions=mod_vlan_vid:16,resubmit(,10)cookie=0x0, duration=1465355.359s, table=2, n_packets=418252, n_bytes=81112777, idle_age=52, hard_age=65534, priority=1,tun_id=0x13 actions=mod_vlan_vid:42,resubmit(,10)cookie=0x0, duration=1631281.273s, table=2, n_packets=445, n_bytes=52848, idle_age=65534, hard_age=65534, priority=1,tun_id=0x17 actions=mod_vlan_vid:37,resubmit(,10)cookie=0x0, duration=2578609.671s, table=2, n_packets=1821, n_bytes=167272, idle_age=16439, hard_age=65534, priority=1,tun_id=0xc actions=mod_vlan_vid:17,resubmit(,10)cookie=0x0, duration=2574619.932s, table=2, n_packets=490592856, n_bytes=279835052124, idle_age=65534, hard_age=65534, priority=1,tun_id=0x19 actions=mod_vlan_vid:19,resubmit(,10)cookie=0x0, duration=2578613.06s, table=2, n_packets=18, n_bytes=756, idle_age=65534, hard_age=65534, priority=1,tun_id=0xe actions=mod_vlan_vid:8,resubmit(,10)cookie=0x0, duration=1469974.534s, table=2, n_packets=6992536, n_bytes=1567235429, idle_age=9, hard_age=65534, priority=1,tun_id=0x7 actions=mod_vlan_vid:41,resubmit(,10)cookie=0x0, duration=2144082.193s, table=2, n_packets=2583, n_bytes=461773, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1d actions=mod_vlan_vid:32,resubmit(,10)cookie=0x0, duration=2578611.169s, table=2, n_packets=4230304, n_bytes=917966422, idle_age=0, hard_age=65534, priority=1,tun_id=0x5 actions=mod_vlan_vid:14,resubmit(,10)cookie=0x0, duration=85135.825s, table=2, n_packets=1739, n_bytes=130092, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1f actions=mod_vlan_vid:53,resubmit(,10)cookie=0x0, duration=979.195s, table=2, n_packets=123, n_bytes=11895, idle_age=933, priority=1,tun_id=0x22 actions=mod_vlan_vid:54,resubmit(,10)cookie=0x0, duration=1898543.732s, table=2, n_packets=240, n_bytes=30712, idle_age=65534, hard_age=65534, priority=1,tun_id=0x16 actions=mod_vlan_vid:34,resubmit(,10)cookie=0x0, duration=2578614.004s, table=2, n_packets=5595775, n_bytes=5465543420, idle_age=4, hard_age=65534, priority=1,tun_id=0x8 actions=mod_vlan_vid:6,resubmit(,10)cookie=0x0, duration=1473941.345s, table=2, n_packets=4202494, n_bytes=2516931444, idle_age=9, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:40,resubmit(,10)cookie=0x0, duration=2578619.787s, table=2, n_packets=103506, n_bytes=13925984, idle_age=0, hard_age=65534, priority=0 actions=drop
wat.
Missing OpenFlow ruleroot@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xbroot@osnet1:~# echo $?1
Try to re-add that network to the responsible agent:
$ neutron dhcp-agent-network-remove 1beb99ef-e6f6-4083-8fb6-661f2f61c565 \ 4dc325ed-f141-41d9-8d0a-4f513defacadRemoved network 4dc325ed-f141-41d9-8d0a-4f513defacad from DHCP agent$ neutron dhcp-agent-network-add 1beb99ef-e6f6-4083-8fb6-661f2f61c565 \ 4dc325ed-f141-41d9-8d0a-4f513defacadAdded network 4dc325ed-f141-41d9-8d0a-4f513defacad to DHCP agentroot@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb cookie=0x0, duration=0.945s, table=2, n_packets=14, n_bytes=588, idle_age=0, priority=1,tun_id=0xb actions=mod_vlan_vid:55,resubmit(,10)
Weirdness #3 - duplicate routers• Intermi)ent connec-vity issues groan
• No DVR or L3-HA enabled
• Routers scheduled and created twice on two network nodes
• Same network configura-on in each namespace
Duplicate routers› neutron l3-agent-list-hosting-router fe79ae7e-debf-44b9-8fd7-601abd5fb928+--------------------------------------+--------+----------------+-------+----------+| id | host | admin_state_up | alive | ha_state |+--------------------------------------+--------+----------------+-------+----------+| 48132c36-b6b1-40fa-b9d9-5474f4f27c3a | osnet0 | True | :-) | || c821a370-b301-40c5-8b7b-25d147ffc904 | osnet1 | True | :-) | |+--------------------------------------+--------+----------------+-------+----------+
› neutron router-show fe79ae7e-debf-44b9-8fd7-601abd5fb928+-----------------------+----------------------------------+| Field | Value |+-----------------------+----------------------------------+| admin_state_up | True || distributed | False || ha | False || status | ACTIVE || tenant_id | 7d718c99276c43d1992d64d061d98f15 |+-----------------------+----------------------------------+
How to approach troubleshoo0ngTroubleshoo*ng checklist
• UUIDs for instance, loca2on, MAC address
• UUIDs for network, subnet, router
• Network node hos2ng L2 and L3 agents
Useful commands - neutron
$ neutron agent-list$ neutron l3-agent-list-hosting-router $router_uuid$ neutron dhcp-agent-list-hosting-net $net_uuid$ neutron router-list-on-l3-agent $agent_uuid$ neutron net-list-on-dhcp-agent $net_uuid$ neutron help
(More) useful commands
Standard network troubleshoo1ng toolkit:
$ tcpdump -enl -i eth1 | grep -i dhcp$ ip netns exec $netns tcpdump port 67 or port 68 -lne$ ip route$ ip address$ iptables-save$ brctl$ mtr
Etc.
Thanks!
Nick JonesDataCentred
h"p://www.datacentred.co.ukh"p://dischord.org
@yankcrime