symerton –using virtualization to accelerate packet processing · 2007-01-30 · title...

Copyright © 2006 Intel Corporation

Symerton – Using Virtualization to Accelerate Packet Processing

Aaron R. KunzeStephen D. Goglin

Erik J. Johnson

Communications Technology LabCorporate Technology Group

December 4, 2006

December 4, 2006 Symerton - Kunze, Goglin, Johnson - ANCS 20062

*Other names and brands may be claimed as the property of others.


S D

Cisco AS5800 SERI ES

Po wer

CISCO SYSTEMS

Complexity at the Network Edge

SD

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

8260

SD

Enterprise LAN

SD

Cisco 1720

BRIS/T

CONSOLE

AUXWIC 0 OK

OK

B2

B1

WIC 1 OK

DSUCPU

LNK100FDX

S3

LOOP

LP

Access Network

MAN/WAN• VPN Gateway• Firewall• Intrusion Detection• XML & SSL acceleration

• L4-L7 switching• Application acceleration

• Compression• Monitoring (billing, QoS)




Problem

Network edge packet processing on a general-purpose OS does not perform well

• Buffer copies

– Required to share network devices between applications...

– ...but network edge applications don’t need to share network devices

• Interrupt-based device management

– Allows processor to stay busy when no packets arriving...

– ...but when no packets are arriving, network edge apps have no work

• Virtual memory

– Allows appearance of more memory than physically available...

– ...but page faults are an eternity for network edge apps

– Allows protection between applications in different trust domains...

– ...but network edge devices are embedded devices with one trust domain (today)




Existing Solutions

Pros

• High performance

• Rich ecosystem of non-performance-critical code

Cons

• High maintenance costs

• License, IP, and upgrade issues

user

kernel

Application (perf-critical)

Heavily-modified off-the-shelf operating

system

Application (non-performance-critical)

= customer developed

= off-the-shelf

Real-time OS

Application

Pros

• High performance

• No need to modify/maintain OS

Cons

• Much smaller ecosystem for skills/code

“The Kernel Hack”

“The RTOS”




Intel® Virtualization Technology Primer

Hardware acceleration for virtual machines (VMs)

Unmodified operating system

App

kernel

user

Virtual machine monitor (VMM)

App App

Unmodified operating system

App

kernel

userApp App

root

non-root

VM enter VM exit VM enter VM exit

kernel

user




Symerton Approach

Special-purpose networking operating system

Application (performance-critical)


No buffer copiesPolled network

interfacesNo paging

= off-the-shelf







= off-the-shelf

Performance Partition

Symerton Approach


Virtual Machine Monitor

non-root

root








= off-the-shelf


Symerton Approach



non-root

root




General-purpose Partition


Off-the-shelf operating system




= off-the-shelf


Symerton Approach



non-root

root







Performance partition maintains dedicated access to network

devices




= off-the-shelf


Symerton Approach



non-root

root







Network driver

Packets passed between performance partition and general-

purpose partition using network driver




What About Virtualization Overhead?

Page faults are more expensive!

• ...we can reduce/eliminate paging

Interrupts are more expensive!

• ...we can use polling

Passing data between VMs is expensive!

• ...only a small fraction of traffic should pass between VMs




Evaluating the Approach

Questions to answer:

• How much performance gain in removing general-purpose OS overheads?

– Are applications faster with no copies, interrupts, paging?

– Is it worth the extra effort?

• How much of that lost due to virtualization?

– Does having an extra software layer erase the gains achieved by resolving general-purpose OS issues?




Application (performance-

critical)

Xen Domain 0Performance Partition

Symerton Proof-of-Concept (POC)

Modified Xen*

XenoLinux*

non-root

root

Modified FreeBSD*

user

kernel

Application (non-performance-critical)user

kernel




Application (performance-

critical)

Xen Domain 0Performance Partition

Symerton Proof-of-Concept (POC)

Modified Xen*

XenoLinux*

non-root

root

Modified FreeBSD*

user

kernel

Xen modified to allow direct access to NIC for

guests with modified drivers

Application (non-performance-critical)user

kernel

FreeBSD modified to poll the NIC and provide zero-

copy access to user space applications

Using Xen means can’t readily turn paging off or

use large pages




POC Evaluation Setup

System Setup

• 3.0GHz Intel® Pentium® D 930 Processor

• Intel® PRO/1000 PF PCI Express (Only one port used)

• Linux* configs are RedHat* Enterprise Linux* 4 Advanced Server Update 2

• FreeBSD configs are FreeBSD 6.0

• Xen 3.0

Two test applications

• Null forwarder (forwards packets)

• Snort (intrusion detection system)

Packet traces

• Traces from real networks, including NLANR traces and Intel IT traces




Null Forwarder on Native and Symerton

0102030405060708090

100

0 200 400 600 800 1000 1200 1400 1600

Packet Size

Per

cent

of M

axim

um L

ine

Rat

e

RedHat FreeBSD

Modified FreeBSD Symerton

Evaluation Results – Packet Forwarding

Large performance increase for small

packets

No noticeable overhead from

Xen/VT

Source: Intel




Evaluation Results – Snort

Snort Throughput w/ Traces

0

20000

40000

60000

80000

100000

120000

1999

0513

1900

1999

0514

1000

1999

0515

0000

AN

L-11

0739

0954

AN

L-11

0740

2013

AN

L-11

0741

3100

MR

A-

1104

7108

88

MR

A-

1104

7219

46

MR

A-

1104

7348

30

plat

o02-

035a

qp

ww

w12

-00

1ggi

ww

w12

-01

5ggi

Trace

Pac

kets

Per

Sec

ond

FreeBSD RedHat SymertonSource: IntelSnort performance improves 22% on average!




Evaluation Results – Snort w/ Non-perfCritical Code

Snort Throughput w/ Traces

0

20000

40000

60000

80000

100000

120000

1999

0513

1900

1999

0514

1000

1999

0515

0000

AN

L-11

0739

0954

AN

L-11

0740

2013

AN

L-11

0741

3100

MR

A-

1104

7108

88

MR

A-

1104

7219

46

MR

A-

1104

7348

30

plat

o02-

035a

qp

ww

w12

-00

1ggi

ww

w12

-01

5ggi

Trace

Pac

kets

Per

Sec

ond

Sym erton Sym erton + Slowpath Sym erton + Slowpath Shared CoreSource: IntelSnort performance drops when VMs share a core




Evaluation Results – Effects of Paging

Can’t turn paging off or use large pages on Xen

When running one particularly bad trace through Snort, observed 374 TLB misses per packet

Using large pages or no paging has potential for more gains

Source: Intel




Conclusions

Currently, designers using mainstream hardware platforms for packet processing face a dilemma:

• High performance of a specialized OS, or

• Rich software ecosystem of a general-purpose OS

Symerton approach offers best of both worlds with low overheads in most cases




Potential Research Topics

Could one design a VMM specifically for this usage model?

• Better support for real-time?

• Compromise some inter-VM security concerns for performance?

What does a special-purpose networking OS look like?

• Better scheduling algorithms?

• Better memory allocation?

• How best to use domain knowledge?

symerton –using virtualization to accelerate packet processing · 2007-01-30 · title...

Documents