vyatta subscription edition 6.5 r1 testing and analysis

RA-111812-02 Vyatta Subscription Edition 6.5 R1 Test Plan and Results November 2012

Testing Initiated Independently by Router Analysis, Inc Copyright © 2012 Router Analysis, Inc.

Vyatta Subscription Edition 6.5 R1Test Plan and Results

Report RA-111812-02November 2012

RA-111812-02 Vyatta Subscription Edition 6.5 R1 Test Plan and Results November 2012

Key PointsVyatta is able to forward 100% Line-Rate 040?�[YHMÄJ�HJYVZZ�HSS�0U[LYMHJLZ�PU�V\Y�test system

Performance degrades as features are added.

Vyatta handles QoS with no issues, WYV[LJ[PUN�[YHMÄJ�L]LU�^OLU�[OL�KLZ[PUH-tion interface is more than 200% over-subscribed.

Vyatta can be run directly on commodity hardware or in a virtual machine

Test System Configuration:

Hardware:Supermicro X9SAE-VIntel I7-3770 / 32GB ECC RAMFour Intel I340-T2 NICs (8 Total Interfaces)Intel 520 Series 240GB SSDCost: ~$1600 US

VM Configuration:VMware 5.1.0 Hypervisor2 Or 4vCPUs, 4GB Of RAMDirect Access To Uplink Ports Via VMDirectpath

Router Analysis Contact [email protected]

Performance Summary:

��-VY^HYKPUN�WLYMVYTHUJL�Y\UUPUN�KPYLJ[S`�VU�OHYK^HYL*VUÄN\YH[PVU!��/;�*HWHISL�*VYLZ��.)�9(4 o 4.2 Mpps at 64 bytes. o Line-Rate at 375 bytes� � � V��4WWZ�040?�[YHMÄJ�^OPSL�KVPUN�(*3Z��\97-�HUK�8V:�

��-VY^HYKPUN�WLYMVYTHUJL�Y\UUPUN�PU�H�]PY[\HS�THJOPUL*VUÄN\YH[PVU!��]*7<��.)�9(4 o Eight Tenants – Two Gigabit Uplinks� � � � � � � �� .IWZ�6\[IV\UK o Line-Rate IMIX between Two Ports o uRPF/ACLs have no impact on performance

Vyatta Subscription Edition 6.5 R1

Testing Dates: October 10th – 30th 2012Report Updated: November 20th 2012

Report Author: Steven Noble

IMIX

Testing Initiated Independently by Router Analysis, Inc Copyright © 2012 Router Analysis, Inc.

RA-111812-02 Vyatta Subscription Edition 6.5 R1 Test Plan and Results November 2012 !

Copyright © 2012 Router Analysis !3

Table of Contents

Table of Contents ................................................................................ 3!Table of Figures ................................................................................... 3!Results Summary ................................................................................. 4!Test Lab Setup .................................................................................... 5!Tests Carried Out ................................................................................. 6!Testing Results Reviewed .................................................................... 7!Appendix A – Test Cases ................................................................... 12!Appendix B – Sample Test Result Data Tables ................................... 14!About Router Analysis ........................................................................ 15!

Table of Figures Figure 1 - Vyatta 6.5 Deployment Scenarios. ..................................... 4 Figure 2 – Vyatta 6.5 R1 VM Setup ..................................................... 5 Figure 3 - Vyatta 6.5 R1 IPv4 and IPv6 IMIX Forwarding .................... 7 Figure 4 - ACL + uRPF Results ........................................................... 8 Figure 5 - QoS Loss On TOS 0 at 57% Line Rate ............................... 9 Figure 6 - QoS Loss on TOS 0 at 57% Line Rate ................................ 9 Figure 7 - At 69% TOS 4 is affected ................................................. 10 Figure 8 - 39% Loss On TOS 4 ......................................................... 10 Figure 9 - 500k BGP Routes ............................................................. 11



Results Summary Vyatta Subscription Edition 6.5 R1 (Vyatta SE 6.5) performed better than

expected when tested in the Router Analysis test lab. Tests were run on both Bare Metal (directly on the server) and in a Virtual Machine (under VMWare ESXi 5.1).

Featureless IPv4 forwarding performance for 64 byte packets crossed

4Mpps after modifications were made to the host server used for the test. Initial non-optimal setup tests showed 1.7Mpps.

Enabling features such as ACLs and uRPF dropped the routers forwarding performance for 64 byte packets down to 3Mpps for IPv4.

QoS functioned as expected, protecting streams as configured. Below are the deployment scenarios that were tested by Router Analysis.

Testing included two different VM setups, one using two vCPUs, one using four vCPUs and a Bare Metal setup using the test lab setup. The following graph is the throughput of each system while running IMIX traffic over it.

Figure 1 - Vyatta 6.5 Deployment Scenarios.

!



Test Lab Setup The Router Analysis test lab utilizes specially built systems from IXSystems

to host software based systems. The hardware specs are the following: SuperMicro X9SAE-V with a single Intel I7-3770 CPU, 32GB ECC RAM,

240GB Intel 520 Series SSD and 4 Intel I340-T2 GbE NICs. Setting up the system to optimally forward packets is a key part of working

with any software forwarding router. The choice of network cards was key to getting the results included in this test.

Vyatta SE 6.5 was configured to use all eight GbE interfaces. Tests were

done with 4GB RAM on Bare Metal but Vyatta never used more than 1GB of ram during any tests.

When testing under VMWare, Vyatta SE 6.5 was given 4GB RAM, 2vCPUs,

vSwitch access to six interfaces and DirectPath access to two interfaces. A total of eight tenant systems were emulated including two on the same VMWare host and six via links to the tester.

To generate data for the 4vCPU test, four VMs were used along with four

GbE interfaces for tenants. The test used four GbE interfaces connected via DirectPath for uplinks.

Figure 2 – Vyatta 6.5 R1 VM Setup



Tests Carried Out

• IP forwarding rate o Minimum Packet Size (64 bytes IPv4, 82 bytes IPv6)

! With ACLs (permit any any) ! With uRPF ! With QoS

o Packet IMIX + Features ! IPv4 ! IPv6

o 1500 Byte Packets + Features • QoS Functionality

o QoS • OSPF

o 5000 Routes ! One Neighbor ! Five Neighbors

• BGP o 500,000 Routes

! One Neighbor ! Five Neighbors

!



Testing Results Reviewed IPv4 and IPv6 IMIX Forwarding Results with Features

Figure 3 - Vyatta 6.5 R1 IPv4 and IPv6 IMIX Forwarding

The main takeaway from the results is that Vyatta SE 6.5 performs well both

in a Bare Metal environment and as a Virtualized Multi-Tenant Router. While testing, the lowest performance appeared while executing the “IPv6

forwarding with ACLs, uRPF, and QoS IMIX” test case. Even then the performance was reasonable at 2.25Mpps.

The biggest impact within virtualization was using the standard vSwitch to

connect hardware interfaces. Future tests may include using the distributed vSwitch or specialized hardware to see how much impact the different vSwitch types have. !

0

500000

1000000

1500000

2000000

2500000

3000000

IPv4 Base IPv4 ACL IPv6 Base IPv6 ACL

2vCPU VM

4vCPU VM

Bare Metal



Base Test Cases: IPv4/IPv6 IMIX Forwarding + Features Tests For these tests we follow the following testing concepts to find the best-case performance of the system. Testing is done using IMIX traffic on bare-metal. Step 1: Configure 8G of ports and send traffic in a full mesh – 2.58Mpps.

Step 2: Configure a simple permit all ACL on the router – No Change. Step 3: Configure the ACL inbound on one interface – 2.481Mpps. Step 4: Configure the ACL outbound on one interface – No Change.

Step 5: Configure the ACL in/out on all interfaces – 2.38Mpps. Step 6: Configure traffic to run at about 2.38Mpps. Step 7: Configure uRPF on one interface – No Change.

Step 8: Configure uRPF on a second interface – No Change. Step 9: Configure uRPF on the rest of the interfaces – No Change. Result: uRPF has no impact on traffic.

!Figure 4 - ACL + uRPF Results

!!



QoS Testing and Results: From our tests we have determined that QoS works as expected on the

system. In a simple test involving four GbE links (a three to one oversubscription), three different TOS bits were set (0, 4 and 7) and traffic was sent balanced (33.3% per TOS) from each GbE towards a single GbE port outbound.

The router was configured so that TOS 0 was protected up to 15% of line

rate, TOS 4 was protected up to 15% of line rate and TOS 7 was protected up to 70% of line rate.

When running the test the following results were seen: < 100% all streams went through without issue > 100% TOS 0 started to be dropped ~ 140% TOS 4 started to be dropped ~ 190% TOS 7 started to be dropped Step 1: Send less than 33% line-rate from each of the three ports

Result: Up to almost 33% line-rate from the 3 ports (< 100% outbound on

other interface) no loss was seen. Above 33% we saw loss for the TOS 0 stream but not the TOS 4 or 7 streams. Step 2: Send 50% line-rate from the TOS 0 port (113% total)

Figure 5 - QoS Loss On TOS 0 at 57% Line Rate

Result: At above 34% we start to see TOS 0 traffic drop as shown in the

magnified section below where we are sending 50% traffic on the TOS 0 link:

Figure 6 - QoS Loss on TOS 0 at 57% Line Rate



Step 3: Up the traffic to about 69% line-rate per port (207% total)

Figure 7 - At 69% TOS 4 is affected

As shown in this magnified section we see about 38% Loss on TOS 4 when sending 69% line-rate per port (207% line-rate outbound)

Figure 8 - 39% Loss On TOS 4

Step 4: Raise the traffic on each stream to above 88.5% traffic loss is seen

on all streams. !! Result: Loss is seen on all 3 QoS traffic profiles. Final Result: QoS works as expected. !

!



OSPF/BGP TESTING AND RESULTS:

For OSPF and BGP we setup the following tests: Test 1: Send 5000 OSPF Routes in on a single interface and verify routes

are forwarding correctly using traffic. Test 2: Send 5000 OSPF Routes in on five interfaces and verify routes are

forwarding correctly using traffic. Test 3: Send 500k BGP Routes in on a single interface and verify routes are

forwarding correctly using traffic. Test 4: Send 500k BGP Routes in on five interfaces and verify routes are

forwarding correctly using traffic. Results: In all cases forwarding worked as expected.

!Figure!9!)!500k!BGP!Routes!

!



Appendix A – Test Cases

The following two test cases represent a skeleton format of the set of tests performed in the Router Analysis lab. Test Case One – IP Forwarding Synopsis: When testing systems that have a set PPS or Forwarding Bandwidth that is lower than the combined interfaces available; it is necessary to validate the best-case base-forwarding rate. Once the rate has been determined, it is possible to judge other test cases against the best-case forwarding rate. Goal of Test: To confirm best-case IPv4 or IPv6 forwarding rate using the minimum and maximum packet size accepted by the system. Procedure: Configure the minimum number of links between the tester and the system under test (SUT) to provide either a.) More than the maximum PPS or b.) More than the maximum forwarding bandwidth as claimed by the vendor. For example: Vyatta SE 6.5 performance is based on the hardware it is installed on. To test the system you would start with the same number of interfaces as cores you have in the system. Data to Col lect: Maximum PPS for IPv4, IPv6 and a 50/50 mix of IPv4 and IPv6 Maximum traffic forwarding for IPv4, IPv6 and a 50/50 mix of IPv4 and IPv6 !



Test Case Two – QoS Testing Synopsis: !This is a simple QoS test based on information from the IXIA BlackBook. We have modified it slightly to utilize more than 2 ports. Goal of Test: Determine if QoS functions as expected by creating three different TOS valued streams, 0, 4 and 7. On the router configure a traffic policy to prefer 7 over 4 and 4 over 0. Procedure: Using IxNetwork create three streams with TOS levels of 0, 4 and 7 on three interfaces going towards a single interface. On the outbound interface configure a QoS policy to prefer TOS 7 over TOS 4 and TOS 4 over TOS 0. Run traffic and slowly raise it until you see packets drop for 0, then 4 and then 7. Addit ive Tests: Add in background traffic from your earlier test where you saw packet loss due to features. Confirm that the QoS policy is still working as expected. Data to Col lect: Packet loss data for each stream at different traffic levels. Screen Captures of the IxNetwork screen during the tests. !



Appendix B – Sample Test Result Data Tables The following tables represent a sample of the data collected during testing in the Router Analysis Lab. IPv4 Forwarding Results Data Bare-Metal Eight GbE Interfaces Packet Size (bytes) Aggregate No Drop Rate (pps) Aggregate TX Rate (Gbps)

64 4017857 2.057 128 3885135 3.978 256 3079710 6.307 512 1879699 7.699 1024 957854 7.846 1280 769231 7.876 1518 650195 7.895

Under VMWare 5.1 Two GbE Interfaces Packet Size (bytes) Aggregate

Throughput (pps) Aggregate Throughput (Mbps)

Aggregate Throughput %

64 500000 256 16.8 128 394321 404 23.3 256 253512 519 28 512 461254 1889 98 1024 239463 1962 100 1280 192307 1969 100 1518 162548 1974 100

!



About Router Analysis

With a world-class fully independent testing lab, Router Analysis provides unbiased information to its clients.

Router Analysis provides the following services: Network Equipment Testing Consultation and Advisement Test Plan Creation On-Site Support Network Design Consultation, Verification and Installation … And More. Contact Router Analysis by phone at +1 888 SONN COM (766-6266) or by

email at [email protected] for a quote, more information or any questions.

vyatta subscription edition 6.5 r1 testing and analysis

Technology

router analysis test

r1 test plan

r1 vyatta se

test lab setup

test cases

testing results

gb intel

r1 ipv4