optimizing nfv performanced2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkspm-2029.pdf · nfv...

Optimizing NFV Performance

Ali Bokhari - Technical Leader

BRKSPM-2029

• Introduction to NFV

• Performance Optimizations

• I/O Optimizations• Network connectivity

• Virtual Switches

• Pass Through

• Compute Optimizations• NUMA

• Huge Pages

• CPU Pinning

• Conclusion

Agenda

Introduction to NFV

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Standards-based Architecture

UCS/Nexus

KVM

OpenStack

OpenStack

Controller

ESC

NSO

Enabled by

Tail-fNew or Existing OSS/BSS

ETSI GS NFV 002 V1.1.1 (2013-10)

NFV Reference

Architectural Framework

EPC Policy

EMEM

IMS

EM

BRKSPM-2029 4


VNF Deployment Options

Bare Metal

• Dedicated hardware

• No virtualization overhead

• High performance

• Easy to secure

Virtual Machines

• Shared host for multiple VNF

• Performance limited by virtualization

• No shared kernel more secure

Containers

• Shared host for multiple VNF

• Bare metal performance

• Shared kernel with host less secure

VNF App

Host kernel

VNF VM

kernel

VNF VM

kernel

Host kernel

VNF

App/OS

VNF

App/OS

Host kernel


Pain Points & Bottlenecks

Network I/O

CPUMemory

Network

Appliances

Virtual

Network

Functions

BRKSPM-2029 6


NFV Performance Evolution

7BRKSPM-2029

Virtual Machines

Virtual Switches

PCI Passthru

SR-IOV

NUMA

Huge Pages

CPU Pinning

High Performance

vSwitch

Containers


Connectivity Options Abound

OVS

OVS

w/DPD

K

Linux

Bridge

Open

Day

Light

Open

Contrail

DPDK

PCI-PT

SR-IOV

PCI-PT

Virtual

switch

Virtual

distributed

switch

VMXNET

Kernel

VMXNET

DPDK

OpenStack VMware

BRKSPM-2029 8


Connectivity in the Cloud

VM VM VM VM

PF

VF VF

VM VM

kerne

l

ov

s

VM VM

kerne

l

ov

s

Virtual Appliance Web-scale

PCI Passthrough

• Entire PCI device passed through to VM

SR-IOV

• PCI device sliced into multiple virtual

devices to be passed to VMs

PCI Passthrough SR-IOV OVS OVS w/DPDK

OVS

• Virtual switch for scale and migration

OVS w/DPDK

• Virtual switch with high speed direct

access to physical NIC

crypto

I/O Optimizations

Network Connectivity


OpenStack Networking – Tenant Networks

OpenStack

Controller

Compute Node Compute Node

L2/L3

Switch

VM VM

Tenant

Networks

• Contained within the cloud

• Access to external networks through the Network Node

• DVR allows L3 agent to reside on each compute node

Network Node

L3DHCP

BRKSPM-2029 12


OpenStack Networking – Provider Networks

OpenStack

Controller

Compute Node Compute Node

VM VM

Provider

Networks

L2/L3 Switch

Network Node

DHCP

BRKSPM-2029 13

Virtual Switches


Virtio-net

Compute Host

User space

Virtual

Machine

• OVS is the default vSwitch in OpenStack

• Neutron makes it simple to use

• OpenStack OVS adds security

• Security Groups

• Lower performance because of large number of VM Exits

• VM Entry: kernel mode to guest mode transition

• VM Exit: guest mode to kernel mode transition

• Since OVS and Linux Bridge run in kernel space, all I/O from the VM (guest space) requires VM exit / entry

eth1

OVS / LB

Kernel

sockets

virtio_net

vNIC

VM 1

QEMU FE

virtqueue

Kernel & Network

Drivers

Tap Device

QEMU BE

Compute Host

Kernel space

BRKSPM-2029 15


vhost-net

• Offers network traffic between the guest and kernel of the host directly

• QEMU is only used for control plane, negotiation, live migration, etc.

igb

eth1

Tap

Device

QEMU BE

QEMU FE

vhost_net

vNIC

virtqueue

Application

Kernel

sockets

Kernel with

OVS Module

Compute

Host

User

Space

Virtual

Machine

Compute

Host

Kernel

Space

BRKSPM-2029 16


What is DPDK?

• Data Plane Development Kit

• Set of data plane libraries and NIC drivers

• Enables a fastpath for high speed packet processing

• Bypasses Linux kernel and network stack

• Can be used on bare metal host or a VM

• Limited security options

Kernel with

Network Drivers

Application Application

DPDK Libraries

Kernel with

Network

Drivers

No DPDK DPDK

user space user space

BRKSPM-2029 17


Open vSwitch with DPDK

• Includes DPDK libraries in OVS

• OVS uses DPDK fastpath network connection

• Allows direct access from the guest to the host’s user space by-passing QEMU

• QEMU is only used for control plane, negotiation, live migration, etc.

• OVS moves into user space – minimizing context switches

• Easier to work with than SR-IOV or PCI-Passthrough

igb

eth1

OVS

with

DPDK

Tap

Device

Application

QEMU BE

QEMU FE

vhost-user

virtqueue

Kernel

sockets

Kernel &

Network

Drivers

Compute

Host

User

Space

Virtual

Machine

Compute

Host

Kernel

Space

BRKSPM-2029 18

Pass Through


PCI-Passthrough Networking

• Pass a PCI device directly to VM

• Limited by number of PCI devices

• VM interface gets full resources of device

• Cumbersome to use

• OpenStack does not see the interface

• Cannot enforce security group rules

• Very good for performance interfaces

eth1

Kernel

sockets

VM 1

vNICVirtual

Machine

Compute Host

User space

Compute Host

Kernel space

BRKSPM-2029 20


Compute Host

Kernel space

SR-IOV

• Singe Root – I/O Virtualization

• Each NIC is a Physical Function (PF)

• Virtualize PFs into multiple Virtual Functions

• VF passed to VM similar to PCI-Passthrough

• VFs share the resources of the PF

• Good for performance interfaces

SR-IOV PF

eth1

Kernel

sockets

Net 2

Virtual Machine 1

Net 1

Kernel

sockets

vNIC

VF Driver

vNIC

VF Driver

VF

Driver

VF

Driver

VF

Driver

Net 1

Kernel

sockets

vNIC

VF Driver

Virtual Machine 2

Compute Host

User space

BRKSPM-2029 21

Compute Optimizations


SMP

• Symmetric Multi-Processing

• All CPUs have equal (uniform) access to shared memory bus

• Does not scale well as the system bus gets congested

CPU 0 CPU 1

I/O Shared Memory

CPU 2 CPU 3

CPU 4 CPU 5

system bus

BRKSPM-2029 23


NUMA

• Non-Uniform Memory Access

• Each CPU in the node has fast access to local memory

• Slower access to remote node’s memory is used less often

• Most of the data used by any CPU will be stored and accessed locally

Local

Memory

system interconnect

Core 1 Core 2

Core 3 Core 4

CPU 0

Core 1 Core 2

Core 3 Core 4

CPU 1

Local

Memory

NUMA Node 0 NUMA Node 1

BRKSPM-2029 24


Huge Pages

• 4K is default page size, but can support 2MB or 1GB

• Larger pages reduce the number of pages, shrinking page table size

• Smaller table size means faster searches

• Larger pages increase cache hits and reduce cache misses

• Can be locked to prevent swapping

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

4K

1GB 1GB 1GB

Huge Page Table

4K Pages 1GB Pages

BRKSPM-2029 25


What is a Virtual CPU?

• 2 Sockets (physical CPUs)

• 8 Cores per Socket

• 2 Threads per Core (hyper-threading)

T1

T2

T1

T2

T1

T2

T1

T2

Core 0 Core 1 Core 2 Core 3

T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


Socket 1 Socket 2

2 x 8 x 2 = 32 vCPU

Server with dual Intel E5-

2690 8-core processors

w/Hyper-threading

BRKSPM-2029 26


CPU Pinning

T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


T1

T2

T1

T2

T1

T2

T1

T2


Socket 1 Socket 2

HypervisorVM

2 vCPUs

VM

4 vCPUs

Compute Host

BRKSPM-2029 27


Combining Optimizations

• Configure NUMA for fast local memory access for CPUs

• Enable huge pages for fast memory searches

• Pin the vCPUs to physical CPU cores/threads

NUMAHuge Pages

CPU Pinning

BRKSPM-2029 28


Summary

Area of

PerformanceScalability

Migration

SupportPerformance Security

Virtio Network Good Yes Low High

PCI

PassthroughNetwork Poor No Very High Low

SR-IOV Network Good No High Low

High Speed

vSwitchNetwork Good Yes High High

Description

NUMA Each NUMA node consists of one or more CPU with high speed access to local memory

Huge Pages Allows for a smaller table of larger memory pages for fast lookup

CPU Pinning CPU resource management to keep applications on dedicated cores

BRKSPM-2029 29


Glossary• DPDK – Data Plane Development Kit

• ESC – Elastic Services Controller

• ETSI – European Telecommunications Standards Institute

• KVM – Kernel-based Virtual Machine

• NFV – Network Function Virtualization

• NFVI – NFV Infrastructure

• NSO – Network Services Orchestrator

• NUMA - Non-uniform Memory Access

• OVS – Open Virtual Switch

• PCI - Peripheral Component Interconnect

• SMP - Symmetric Multi-Processing

• SR-IOV – Single Root I/O Virtualization

• VNF – Virtual Network Function

• VNFM – VNF Manager

BRKSPM-2029 30


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

BRKSPM-2029 31

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


US Ciscolive 2016 @ Las Vegas- SP Mobility SessionsSession ID Time Session Title Speakers

Sunday (10 July 2016)

LTRSPM-2022 01:00 – 05:00 PM Virtual Packet Core Orchestration: Spring to Life Gateways and Services Julie Ann Connary, Amir Ahmadi

Monday (11 July 2016)

BRKSPM-2125 08:00 - 10:00 AM Virtualizing Cisco Mobile Packet Core Aeneas Dodd-Noble, Vivek Agarwal

BRKSPM-2071 01:00 - 03:00 PM 5G Technology Updates Paul Polakos, Prakash Suthar

BRKSPM-2029 04:00 – 05:30 PM Optimizing NFV Performance Ali Bokhari

BRKSPM-2022 04:00 – 05:30 PM Using Telco NFV to Deploy Mobility Networks Dave Clough

BRKSPM-2130 04:00 – 05:30 PM Mobile Packet Core (ASR5500) Troubleshooting Guilherme Correia, Rama Ramachandran

Tuesday (12 July 2016)

BRKSPM-2128 08:00 - 10:00 AM Radio Access Network Virtualization Mark Grayson, Oliver Bull

BRKSPM-2026 01:30 - 03:00 PM AT&T Case study for Managing Customer Experience in Mobile Networks Jeronimo Diez De Sollano, Anwin Kallumpurath

BRKSPM-2065 04:00 – 05:30 PM Mobile and Enterprise Security for the Age of Ubiquitous Encryption Humberto La Roche, Christopher ORourke

BRKSPM-2127 04:00 – 05:30 PM Design & Deploying Trusted and Un-Trusted VoWiFi Venkata Reddy Kasu, Arun Gunasekaran

Wednesday (13 July 2016)

BRKSPM-2129 08:00 - 10:00 AM Deploying 4G/LTE branch routers for IOT solutions Vivek Datar, David Mindel

PLNSPM-2021 01:30 – 03:00 PM Network Infrastructure and Operation Transformation Anand Malani, Scott Clark, Vilma Stoss, Rich Plane, Mark Ghattas (Moderator)

BRKSPM-2028 04:00 – 05:30 PM Design and Deployment of Overlay Services on LTE Network Tom Redman

BRKSPM-2122 04:00 – 05:30 PM Virtualizing Mobile Service Providers Transport Network Ravi Narahari, Abbas Abidi

Thursday (14 July 2016)

BRKSPM-2121 08:00 - 10:00 AM Cisco SON and Advanced Analytics Oliver Bull, Ashish Bansal

BRKSPM-2126 04:00 – 05:30 PM Analytics for Large Connected Venues: App Integration with Wi-Fi Infrastructure Matt Swartz, Joshua Suhr

PSOSPM-2505 01:00 – 02:00 PM Cisco Hosted Mobility (Mobility as a Service) Mike Shomaker

WISP Lab (11-14 July)

LABSPM-2011 10:00 - 6:00 PM Cisco Virtualized Packet Core Installation on VMware Amir Ahmadi, William Pedraza

LABSPM-2012 10:00 - 6:00 PM Cisco Virtualized Packet Core Installation on Openstack Christopher Ove, Guilherme Correia

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33Presentation IDPresentation ID

Please join us for the Service Provider InnovationTalk featuring:

Yvette Kanouff | Senior Vice President and General Manager, SP Business

Joe Cozzolino | Senior Vice President, Cisco Services

Thursday, July 14th, 2016

In the Oceanside A room

What to expect from this innovation talk

• Insights on market trends and forecasts

• Preview of key technologies and capabilities

• Innovative demonstrations of the latest and greatest products

• Better understanding of how Cisco can help you succeed

Register to attend the session live now or

watch the broadcast on cisco.com


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

BRKSPM-2029 34

Thank you

optimizing nfv performanced2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkspm-2029.pdf · nfv...

Documents