moongen - a scriptable high-speed packet generator · a scriptable high-speed packet generator paul...

Chair for Network Architectures and Services Technical University of Munich (TUM)

MoonGenA Scriptable High-Speed Packet Generator

Paul Emmerich

January 31st, 2016FOSDEM 2016

Chair for Network Architectures and ServicesDepartment of Informatics

Technical University of Munich (TUM)

Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 1


Outline

Hardware vs. Software Packet Generators

Architecture of MoonGen

Hardware Timestamping on Commodity NICs

Precise Rate Control

Example Measurements



Source: www.spirent.com


www.spirent.com


Challenges for software packet generators

I Hardware packet generators areI PreciseI AccurateI Fast

I Software packet generatorsI Run on cheap commodity hardwareI Flexible

I Key challenges for software packet generatorsI Rate controlI Timestamping



Design goals

Design goal of MoonGenCombine the advantages of both approaches while avoidingtheir disadvantages.

I Fast: DPDK for packet I/O, explicit multi-core support

I Flexible: Craft all packets in user-controlled Lua scripts

I Timestamping: Utilize hardware features found on moderncommodity NICs

I Rate control: Hardware features and a novel softwareapproach



Architecture

MoonGen Core

DPDK

Use

rscr

ipt

Mo

onG

enH

W NIC NIC

Port

Q0 ... Qn

Port

Userscript

Lua VM

Userscriptspawn

Userscriptslave

Lua VM

Userscriptmaster

Lua VM

config API data API

config API data API



Hardware timestamping

I NICs support PTP for precise clock synchronization

I PTP support requires hardware timestamping capabilities

I These can be (mis-)used for delay measurementsI Typical precision

I ±6.4 ns (Intel 10 GbE chips)I ±32 ns (Intel GbE chips)

I Some restrictionsI Packets must be UDP or PTP L2 protocolI Minimum UDP packet size is 84 bytes



Software rate control in existing packet generators

Loadgen

NIC

DuT

NICp5

p5 p4 p3 p2 p1 p0

Qmemory QNIC Wire

I Software tries to push single packets to the NIC

I Queues cannot be used, no batch processing

I NICs work with an asynchronous push-pull model

I This can lead to micro-bursts

: Unreliable, imprecise, and bad performance



Hardware rate control

Loadgen DuT

NICNICp9

p9 p8 p7 p6 p5 p4 p3 p2 p1 p0

HW rate controlenabled

Qmemory QNIC Wire

I Modern NICs support rate control in hardware

I Limited to constant bit rate and bursty traffic

I Precision controlled by the hardware

: High performance as queues can be used, but inflexible



Software rate control based on invalid packets

Loadgen DuT

NICNICp6

p6 pi3p5pi

4 pi0p2pi

1p3pi2p4 p1 p0

HW rate controldisabled

p5

Qmemory QNIC Wire

I Fill gaps with invalid packets pi (e.g. bad CRC)

I NIC in the DuT drops invalid packets without side-effects

I Combines advantages of both approaches

I Precision limited by byte rate (0.8 ns per byte) and minimumpacket size (33 byte)

: High performance & high precision



Does it work?

I Test setup: forward packets with Open vSwitchI Measure the latency of the device under test

0 0.5 1 1.5 20

20

40

60

80

100

120

140

160

Offered load [Mpps]

Lat

ency

[µs]

CBR Hardware (Median)

CBR Hardware (25/75th perc.)



Does it work?

I Compare both rate control approachesI Maximum deviation: 2%

0 0.5 1 1.5 20

20

40

60

80

100

120

140

160

Offered load [Mpps]

Lat

ency

[µs]

CBR Hardware (Median)

CBR Hardware (25/75th perc.)

CBR MoonGen (Median)

CBR MoonGen(25/75th perc.)



Does it matter?

I Compare CBR with Poisson trafficI Different response from the device under test

0 0.5 1 1.5 20

20

40

60

80

100

120

140

160

Offered load [Mpps]

Lat

ency

[µs]

CBR MoonGen (Median)

CBR MoonGen (25/75th perc.)

Poisson MoonGen (Median)

Poisson MoonGen (25/75th perc.)



Example: Linux NAPI

I Open vSwitch on Linux

I Uniform distribution caused by interrupt throttling

0 10 20 30 40 50 60 700

0.2

0.4

0.6

0.8

Latency [µs]

Pro

bab

ility

[%]



Example: Linux Virtualization (VirtIO)

I Open vSwitch forwarding through a VM

I Long tail distribution, typical for VMs

100 150 200 250 300 3500

0.1

0.2

0.3

Latency [µs]

Pro

bab

ility

[%]



Example: Hardware Switch

I AS5712-54X 10/40 GbE OpenFlow switchI Bimodal distribution caused by more input than output ports

I Some packets are forwarded directly (cut-through switch)I Some packets are blocked by another flow and buffered

1 1.5 2 2.5 3 3.5 40

2

4

6

Latency [µs]

Rel

ativ

eP

rob

abili

ty[%

]



Summary

I Speeds of 10 Gbit/s per CPU core (64 byte packets)

I Sub-microsecond precision and accuracy

I Execute user-defined script code for each packet

I Easy to use



Q & A

Try MoonGen yourself!

https://github.com/emmericp/MoonGen

Questions?Paul Emmerich – MoonGen: A Scriptable High-Speed Packet Generator 18




[Backup slide] Performance I: Lua can be faster than C

I UDP packets from varying source IP addresses

1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2

5

10

15

CPU frequency [GHz]

Pac

ket

rate

[Mp

ps]

MoonGenPktgen-DPDK

I Pktgen-DPDK needs a complicated main loop that covers allpossibilites

I MoonGen can use a tight inner loop



[Backup slide] Performance II: heavy workload and multi-core scaling

I Generate random UDP packets on 2 10 Gbit NICs

I 8 calls to Lua’s standard math.random per packet

I CPUs artificially clocked down to 1.2 GHz

1 2 3 4 5 6 7 80

10

20

30

Number of 1.2 GHz CPU cores

Pac

ket

rate

[Mp

ps]



[Backup slide] Performance III: 40 GbE

I Generate random UDP packets on 2 10 Gbit NICs

I 8 calls to Lua’s standard math.random per packet

I CPUs artificially clocked down to 1.2 GHz

64 96 128 160 192 224 2560

10

20

30

40

50

Packet size [byte]

Rate[Gbit/s]

1 core 2 cores 3 cores



[Backup Slide] Rate control: 500 kpps

0

40

MoonGen (hardware)

0

20

Pro

bab

ility

[%]

Pktgen-DPDK

0.5 1 1.5 2 2.5 3 3.5 40

20

Inter-arrival time [µs]

zsend



[Backup Slide] Rate control: 1,000 kpps

0

20

MoonGen (hardware)

0

20

Pro

bab

ility

[%]

Pktgen-DPDK

0.5 1 1.5 20

40

Inter-arrival time [µs]

zsend



[Backup Slide] HW/SW rate control details

Rate Software Bursts ±64 ns ±128 ns ±256 ns ±512 ns

MoonGen 0.02% 49.9% 74.9% 99.8% 99.8%500 kpps Pktgen-DPDK 0.01% 37.7% 72.3% 92% 94.5%

zsend 28.6% 3.9% 5.4% 6.4% 13.8%

MoonGen 1.2% 50.5% 52% 97% 100%1000 kpps Pktgen-DPDK 14.2% 36.7% 58% 70.6% 95.9%

zsend 52% 4.6% 7.9% 24.2% 88.1%



[Backup slide] Effects of bad rate control

I Interrupt rate of an Open vSwitch packet forwarder

0 0.5 1 1.5 20

0.5

1

1.5

·105

Offered load [Mpps]

Inte

rru

pt

rate

[Hz] Load generated with MoonGen

Load generated with zsend

I Micro-bursts confuse dynamic interrupt throttling

I This affects latency (cannot be measured with zsend)



[Backup slide] Hardware timestamping precision and accuracy

I Measure latencies of cables of various length

I Calculate encoding time k and propagation speed vp

0 2 4 6 8 10 12 14 16 18 20 220

100

200

300

400

Cable length [m]

Tim

e[ns]

Latency

I Result for fiber cable: k ≈ 311ns, vp = 0.72c ± 0.056c



[Backup slide] Effects of invalid packets

I Median latency of an Open vSwitch packet forwarder

I Packet rate controlled by hardware vs. invalid frames

0 0.5 1 1.5 2−4

−20

2

4

Offered load [Mpps]

Dev

iati

on[%

]

I Minor modifications to the DuT (e.g. an active SSH session)result in a deviation of up to 15% with the same rate controlmechanism


moongen - a scriptable high-speed packet generator · a scriptable high-speed packet generator paul...

Documents