presenter vivekanandan narasimhanfiles.meetup.com/3498272/4. openstack-neutron-dvr... · 2015. 6....

PresenterVivekanandan Narasimhan

Distributed Virtual RoutersOpenstack India Meetup

Agenda

Introduction

High level architecture and DVR Configuration

East-West Routing

Distributed Virtual Routers Openstack India Meetup

Performance

North-South Routing

Services

Legacy Routing in NeutronNetwork node provides:

• IP forwarding

– Inter-subnet (east-west) traffic between VMs

– Floating IP (north-south) traffic between external and VM

– Default SNAT (north-south) traffic from VM to external

• Metadata Agent

– access to Nova metadata service

Issues:

• Performance bottleneck

• Scalability limitations

• Single Point of Failure

Compute node

Network node

Compute node

VM1 VM2

VM1 VM2

Virtual

Physical

Router A

RtrA

OpenStack India MeetupDistributed Virtual Routers

Distributed Routing in NeutronCompute node provides:

• IP forwarding for local VMs

– Inter-subnet (east-west) traffic between VMs

– Floating IP (north-south) traffic between external and VM

• Metadata Agent for local VMs

– access to Nova metadata service

Advantages:

• Bypass network node improves performance

• Scales with size of compute farm

• Limited failure domain (per compute node)

Limitations:

• Default SNAT function is still centralized

Compute node

Network node

Compute node

VM1 VM2

VM1 VM2

Virtual

Physical

Router A

RtrADefault

SNATRtrA’’RtrA’

Distributed Virtual Routers OpenStack® India Meetup

High level requirements for DVR

• Help close the parity gap with Nova (multi-host)

• Provider feature

– Tenants should not have to know or care

• Configurable on a per-router basis (centralized or distributed)

– Default router type set by a global config knob

• Can be deployed into existing environments

• Centralized routers and Distributed routers can coexist in same cloud

• Ability to migrate a router from Centralized to Distributed

• Minimize overhead use of public IP addresses

• Leverage existing code base

OpenStack® India MeetupDistributed Virtual Routers

DVR High Level Architecture and Configuration

Distributed Virtual Routers OpenStack India Meetup

Network/Service Node(s)

Compute Node(s)

VMVM

VM

l3-agentovs-agent

Controller Node

Neutron controller/plugin L3 Plugin

Network/Service Node(s)

l3-agentovs-agent

dhcp-agent qr

qrqr

agent_mode=dvr

router_distributed=True

enable_distributed_routing=Trueagent_mode=dvr_snat

enable_distributed_routing=True

• One FIP namespace supports multiple routers

• Legacy OpenStack Routers (non-distributed) can exist on the network/service node

• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.

FIP

SNAT

Distributed Virtual Routers

metadata

metadata

qrqr

DVR High level Architecture

OpenStack India Meetup

DVR - Features

The DVR VLAN effort in Kilo enabled the following:

DVR can route across two VLAN networks

DVR can route between a VLAN network and VXLAN network

DVR can route between a VLAN network and GRE network

Retained from Juno:

DVR continue to route between two VXLAN networks

DVR continue to route between two GRE networks


“neutron.conf”

router_distributed=True

Plugin Configuration ( Default Router Option)

“l3_agent.ini”

agent_mode=dvr_snat ( Network/Service Node)

agent_mode=dvr (Compute Node only)

agent_mode=legacy ( Network/Service Node)

router_delete_namespaces=True ( Enable namespace

cleanup)(Default)

L3 Agent Configuration and Modes

DVR Configuration Overview


“ovs_neutron_plugin.ini”

enable_distributed_routing = True

enable_tunneling = True

local_ip = <Local Data Network IP (or) TEP IP>

tunnel_types = vxlan

l2_population=True

L2/OVS Agent Configuration

DevStack Configuration

“local.conf”

Q_DVR_MODE=dvr_snat/dvr/legacy (Default)

DVR Configuration Overview


East-West Routing


L3 Agent

Rules------------

Rules------------

vm1 vm2

br-int

br-tun

qr

LMAC

East-West routing is accomplished through a combination of the following in each CN:

•qr : a namespace that forwards traffic among routed subnets using route table entries.

•LMAC: a MAC address unique to each CN for use as source address in forwarding routed frames from one CN to another.

•A set of rules in the OVS bridges that prevent frames with src=gateway MAC from egressed tunnel bridge or the IR. These rules also cause peer bridges to substitute/restore gateway MACs with LMACs while routed frames traverse

through br-tuns and underlay network.

br-ex

Compute Node entities for enabling the DVR East West Routing

L3 Agent


vm1

br-int-cn1

br-tun-cn1

qr

vm2

br-int-cn2

br-tun-cn2

qr

Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN NetCN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1

Data Network

CN1 CN2

PING REQUEST from vm1 to vm2

vm1 ipqr red macvm1 mac vm2 ip

vm1 ipqr red mac

vm2 ip

vm1 ipvm2 macqr grn mac vm2 ip

vm1 ipvm2 macl_mac vm2 ip

vm1 ipvm2 macl_mac vm2 ip

1

2

3

4

5

grn vni

grn vlan

vm1 ipvm2 macqr grn mac vm2 ip6

vm1 mac

Flow of DVR routed packets (example)


North-South Routing


Overview of North-South Routing

External Network

vm1 vm3

br-int

br-tun

vm5

br-ex

• IP Rules

•Routing Entries

• Iptable based NATing rules

• /32 addresses for the floating IPs

• One Floating IP Namespace per CN

• Interface on external network

• Proxy ARPs enabled for hosted

floating IPs

qr

FIP

L3 Agent

Data Network


Distributed Virtual Routers

Default SNAT Traffic flow

OpenStack India Meetup

Services


Services support with DVR

Services Support in DVR as of Kilo

LBaaS

FWaaS• North-South only ( External Traffic)

MetadataService

VPNaaS support for DVR .


Performance


• North/South Traffic for FIPs: VMs have direct access to external network, bypassing the “network node”

• East/West Traffic: VMs which exists in the same hypervisor see the most performance gain. VMs in different hypervisor see some gain also.

Performance benefits of DVR


Hypervisor

Hypervisor

Hypervisor

Hypervisor

vms

vms

vmsvms

NetworkNode

Centralized Virtual Router(CVR)

Physical Switch

Outside workd(ext-net

DVR East/West Traffic

DVR North/South FIP Traffic

CVR/legacy East/West Traffic

CVR/legacy North/South FIP Traffic

Traffic flow differences

Hypervisor

Hypervisor

Hypervisor

Hypervisor

vms

vms

vmsvms

NetworkNode

Centralized Virtual Router

(CVR/legacy)Physical Switch

Hypervisor

Hypervisor

Hypervisor

Hypervisor

vms

vms

vmsvms

NetworkNode

Distributed Virtual Routing(DVR)

Physical Switch

Hypervisor

Hypervisor

Hypervisor

Hypervisorvms

vms

vmsvms

NetworkNode

WITH DVR Enabled

Outside world(internet for example)

Outside world(internet for example)

External Gateway

External Gateway

Physical Switch

IR

External Gateway


Vms inDifferentSubnets

Vms in DifferentSubnets

NORTH/SOUTH

Simultaneous Individual

Mbps Mbps %of Max

dvr_vm_west 5272 9283 99%

dvr_vm_east 2123 8972 96%

cvr_vm_west 1168 1939 21%

cvr_vm_east 809 2083 22%

Total(Max) 9372

netperf -l 20 -H 192.0.2.10EAST/WEST

Simultaneous Flows

From To Mbps Aggregate

dvr_vm_west dvr_vm_east 4858

dvr_vm_east dvr_vm_west 5156 10014

cvr_vm_west cvr_vm_east 576

cvr_vm_east cvr_vm_west 431 1007

netperf -l 20 -H <to_vm>

Some quick test results


THANK YOU


Questions?


presenter vivekanandan narasimhanfiles.meetup.com/3498272/4. openstack-neutron-dvr... · 2015. 6....

Documents