enabling dpdk/sr -iov for · • make a single pcie ethernet controller (pf) to appear as multiple...
TRANSCRIPT
Enabling DPDK/SR-IOV for containerized Virtual Network Functions with Zun
Bin Zhou [NFV Researcher, Lenovo]Hongbin Lu [Zun PTL,Huawei]Yaguang Tang [NFV Researcher, Lenovo]Shunli Zhou [Zun Core, Fiberhome]
November 2017
➡Introduction to Zun➡Zun Container for NFV
• Challenges & Gaps• SR-IOV support in Zun• Container with DPDK➡Performance Benchmark Tes ting
• Setup• Results➡Demo➡Conclus ion
Agenda
Which Emerging Technologies Interest OpenStack Users?● Containers are the
most interesting emerging technologies.
● 75% of OpenStack users interests in containers.
➡How to use containers on OpenStack?➡Exis ting solutions
• Integrate containers into Nova• Example: Nova-docker, Nova-lxd
• Ins tall Container Orches tration Engine (COEs) on VMs.• Example: Magnum, Kubespray
• OpenStack Container service: Zun
Introduce Zun
● OpenStack Container service● Provide API for provisioning and
managing containers without VMs○ Speed○ Simplicity
● Arbitrary memory and vCPUs● Containers as first class resource○ Keystone RBAC for individual
container○ Neutron port(s) for each container○ Cinder volume(s) bind-mount
Introduce Zun
VMs ContainersCreateListDelete
RunExec...
SSHMigrate...
Nova Zun
➡Nova-docker• Use Nova to manage
containers• Suitable if VMs and
containers are the same➡Obstacles
• VMs and containers are different
• Container specified features are not exposed
Introduce Zun
Baremetal
Tenant 1
Virtualization
Tenant 2 Tenant 3
COE
Baremetal
Tenant 1
Virtualization (optional)
Tenant 2 Tenant 3
Containers
ZunCOE COE
Containers
Containers
Containers
Containers
Containers
Magnum Zun
➡Magnum• Provis ion Nova ins tances• Ins tall a COE• Run containers on the
COE➡Pros :
• Strong Isolation➡Cons:
• Low resource utilization• Virtualization penalty
Introduce Zun
➡Concepts:• Container: A s ingle container
• create, update, delete, s tart, s top, kill, …• network-attach, add-security-group, …• attach, exec, commit, log, ...
• Capsule (Experimental): A group of containers that are co-located, have shared network and volumes .• create, lis t, delete, …
Introduce Zun
Introduce Zun
➡Zun API• Provide REST APIs• Manage all compute nodes• Scheduling containers➡Zun Compute
• Compute node agent• Manage local containers• Track compute resources➡Kuryr
• Bind neutron ports to containers
Zun API Zun Compute Docker
Keystone
KuryrNeutron
Cinder
➡Introduction to Zun➡Zun Container for NFV
• Challenges & Gaps• SR-IOV support in Zun• Container with DPDK➡Performance Benchmark Tes ting
• Setup• Results➡Demo➡Conclus ion
Agenda
➡What is NFV• A new way to des ign, deploy and manage network services• Replace hardware with software• Move network functions to commodity hardware➡Benefits of NFV
• Fas t provis ioning • Quick scale up and down• Easy upgrade and relocate• Reduce cos t• No vendor hardware locked-in
Container for NFV
➡VM or Containers?• Time to provis ion: container boots fas ter
• Resource consumption: container has less memory footprint
• Package management: Docker makes it easy
• Configurability: container is better
• Portability: container image is smaller
• Security: VM provides better isolation
• Use Clear Container to improve security
Container for NFV
Challenges & Gaps of using containers
NFV Req features VM Container
SR-IOV Yes Weak
DPDK Yes Weak
CPU pinning Yes Weak
NUMA Yes Weak
Hugepage Yes Weak
➡Lack of supports of NFV required features in container ecosys tem• Container runtime• Container orches tration• OpenStack integration➡Use Zun to reduce the gaps
Enable SR-IOV in Zun➡What is SR-IOV?
• A standardized mechanism to virtualize PCIe devices• Make a single PCIe Ethernet controller (PF) to appear as multiple PCIe
devices (VF)• PF: Physical Function• VF: Virtual Function
• Passthrough VF to container• Bypass virtual switch layer
Enable SR-IOV in Zun➡Enable SR-IOV in Zun
• Create VFs in compute nodes• Configure Neutron• Configure Zun
• Whitelist PCI devices (e.g. pci_passthrough_whitelist = { "devname": "eth3", "physical_network": "physnet2"})
• Enable PCI filters (e.g. enabled_filters = ...,PciPassthroughFilter)• Configure Kuryr
• Enable SR-IOV driver
Enable SR-IOV in Zun1.Create a SR-IOV port2.Create a container3.Pick a host that has available
VFs4.Assign a VF to the port5.Create a container6.Docker calls its network plugin
(Kuryr) to setup the network7.Kuryr retrieve VF’s information
from the neutron port and perform port binding
Zun API
Zun Compute
Kuryr
Neutron
Docker
User
1
2
3
5
6
7
4
➡Introduction to Zun➡Zun Container for NFV
• Challenges & Gaps• SR-IOV support in Zun• Container with DPDK➡Performance Benchmark Tes ting
• Setup• Results➡Demo➡Conclus ion
Agenda
Container with DPDK
DPDK PMD
● physical nic
○ igb_uio
○ vfio-pci
● virtual hardware
○ virtio_user vhost
software
● net_pcap (kernel stack)
Host kernel
Container Container
VF VFPF
PF driverHost kernel
ContainerDPDK DPDK
DPDK
DPDK & SR-IOV for container
SR-IOV in userland
SR-IOV in kernel
VFVF
VF driver VF driver
Container
netns
ETHx
netns
ETHx
Passthrough
Case 1 (non DPDK)
● Zun Container with SR-IOV
● Zun Container with OVS
networking
Performance Benchmark TestingCase 2
● Container with SR-IOV & DPDK (kernel land)
● Container with SR-IOV & DPDK (user land)
Role Hardware OS network CPU
Controller Think system x3650 M5 Ubuntu 16.04.3
82599ES 10Gb
Intel(R) E5-2680 v3 @ 2.50GHz
compute Think system x3650 M5 Ubuntu 16.04.3
82599ES 10Gb
Intel(R) E5-2680 v3 @ 2.50GHz
Software version other
DPDK 17.05
Openvswitch 2.8.1
Testing setup
● L2FWD as containerized
VNF
● RFC 2544 standard
throughput testing
● DPDK-pktgen as packet
generator
DPDK Testing
non DPDK Testing● iperf3 with udp
zun-compute
Server1
zun-compute
Server2
OVS
OVS
container
container
container
container
Linuxbridge Linux
bridge
PF PF
Zun networking without SR-IOV
zun-compute
Server1
zun-compute
Server2
container
container
container
container
VF
VF
VF
VF
Zun networking with SR-IOV
Container network Benchmarking
● Hugepage size
● PCIe NUMA
● Isolate CPU cores for tx/rx pktgen
● Disable isolated cpu core interrupts
BOOT_IMAGE=/vmlinuz-4.4.0-87-generic root=/dev/mapper/docker2--vg-root ro default_hugepagesz=1G hugepagesz=2M hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on isolcpus=5,6,7,8,9,10 nohz=on nohz_full=5,6,7,8,9,10 rcu_nocbs=5,6,7,8,9,10
DPDK testing tuning
Server1 Server2
VF1
VF2
pktgen VNF l2fwd
VF1
VF2
VF1
Testing scenario 1 ● Userland SR-IOV used by container● DPDK application l2fwd inside container
Container
dpdk-devbind --bind=igb_uio 0000:06:10.2
docker run -v /dev/hugepages/:/dev/hugepages --net=none --privileged --name test2 -dit 14ce48b74dd9
l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket-mem 1024,1024 -- -q 8 -p 1
Server1 Server2
VF1
VF2
pktgenVNF
l2fwd
VF1
VF2
VF1
Testing scenario 2 ● containers using SR-IOV by kernel netns● DPDK application l2fwd inside container
NETNS
Container
$ neutron port-create sriov --name sriov_port --binding:vnic_type direct$ zun run --net port=sriov_port dpdk-test
l2fwd -l 5-6 -n 4 --huge-dir /dev/hugepages --socket-mem 1024,1024 --vdev=’eth_pcap0,iface=eth0’ ---q 8 -p 1
Container DPDK Benchmarking
SR-IOV & DPDK can accelerate container networking performance
Benefits
High throughput Low latency Deterministic networking
Conclusion
● DPDK & SR-IOV for container user land approaching physical
server performance
● multi-tenancy issue
● security issue
● Container with SR-IOV for
high throughput non DPDK application
● unified management of VF
@OpenStack
Q&AThank you!
openstack openstack OpenStackFoundation