slb perf automation

SLB PERF AUTOMATION

Agenda

• Perf Tenants Overview • Perf Test cases• Perf Setup• Performance Test Metrics• Investigation Perf Bugs

Perf Tenants• Client Tenants ( Generates Inbound Traffic )

• Queries Xstore configuration table (SLBPerfClientConfigTable) for TenantName mentioned in its service definition file periodically.

• Parses the Xml configuration see below and starts the http/udp/tcp traffic • Xml configuration consists of

– Servers endpoints to send traffic– No of clients ( threads) that will send traffic– MessageSize,frequency,url to target, sendtime, resttime .. Etc

• Log client traffic data ( pass/fail request ) to the Xstore table mentioned in the configuration

• Server Tenants ( Generates Outbound Traffic )• Based on Http/Tcp/Udp, listens on the endpoint for incoming requests from the client• Same as Client tenant but reads (SLBPerfServerConfigTable) for TenantName mentioned in its service definition

file periodically. • Parses the Xml configuration see below and starts the http/udp/tcp outbound traffic

Sr Tenant Type Artifact No Of Tenants

No Of DIPS

No OfEndpoints

No Of Direct Ports

Total Enpoints

1 Http tenant sh25EP6DP1DIP 150 1 25 6 4650






7 Http tenant Private25EP100DP10DIP 1 10 25 100 1025

10 Http tenant WebServer400EP1DIPXS_1 1 1 400 0 400

Total Endpoints: 16485

Server Cluster (CO1CNFPerf01) -with 3 replica• Deployed with SLB to be tested • Deployed with Canary tenant (for inbound failures /Outbound ) • Deployed with ~400- 500 server tenants of varying configurations ( varying

endpoints, dips, protocols ( udp/http ) ). Client Cluster

• Deployed with 20 clients (http/udp

Server cluster load• Mux Throughput ~800 Mbits/sec• HP SNAT per sec = 8 • No of endpoints in the cluster ( see tenant configurations) - ~15,000

Perf Setup

Performance Testing Overview Process

XStore

Server ClusterClient Cluster

Client tenant_2

Client tenant_n

Client tenant_1

Read configurations and upload results

Servertenant_1

Servertenant_2

Servertenant_n

Read configurations and upload results

Outbound traffic

In-bound traffic

CanaryTenant

• Data Plane

– Measure Inbound and outbound Connectivity (Success and Failure rate)– Measure Latency

Sr. no Scenario From => to ( time in UTC) Connectivity data

1 SLBM update

2/25/2013 7:12:10 AM => 2/25/2013 7:43:01 AM Inbound: Success Percentage: 100% Average Latency time: 0.1 Seconds

Outbound: Success Percentage: 100% Average Latency time: 8.66 Seconds

2 MUX update


Outbound: Success Percentage: 99.85% Average Latency time: 9.22 Seconds

3 SLB HP update:


Outbound: Success Percentage: 99.99% Average Latency time: 5.31 Seconds

Performance Test Metrics

• Control Plane

– Programming time • SLBM only failure• LBE failure• SLB + LBE failure

– Deployment of various tenants

Performance Test Metrics ( Cont …)

Serial No. Test-Actions. Programming Time (SLB 9) Result Comments

1 SLBM only < 1 min Pass

2 LB-Engine restart < 1 min Pass No Goal states resent

3LB-Engine restart + SLBM failover

597.77 seconds. Pass

• SLBM Counters• No Failover of slbm for 3 days • No memory build up• CPU hovering around 30% (expected for 80 K snat slots)• HP’s connected to SLBM stable

• Mux Counters• No Mux failures in the last 3 days• No memory build up• CPU on the 1 core (data path) is 28%

• Canary failures– https://test1.diagnostics.monitoring.core.windows.net/content/

search/search.html?table=FcCanaryMeasurementEtwTableVer1v0&start=2013-02-24+05%3a53%3a29Z&end=2013-02-25+08%3a01%3a35Z&query=%22cluster%3d%3d%22%22CO1CNFPerf01%22%22+and+IsError%3d%3dTrue%22%3b+groupby+Analyzer+&useindex=True&utc=True

Performance Test Metrics ( Cont …)

Perf test cases• http://team/sites/cnftest/_layouts/xlviewer.aspx?id=/sites/cnftest/Shared%2

0Documents/SLBPerfTestcases.xlsx

http://team/sites/cnftest/_layouts/xlviewer.aspx?id=/sites/cnftest/Shared%20Documents/SLBPerfTestcases.xlsx

http://team/sites/cnftest/_layouts/xlviewer.aspx?id=/sites/cnftest/Shared%20Documents/SLBPerfTestcases.xlsx

Perf Improvement Plans• Currently setting pre-conditions for Mux Throughput and HP SNAT are achieved is

semi-manual process ( feed back loop) ( Xstore settings changes ). – Need tools for above

• More granular analysis and measurements– Checking failure on every clients– Latency for different sizes of upload and download.

Tools / Resources

• Tcping64• SLB Perf Test Automation path

o ..\rd_cloudnet_n_slb\services\cloudnet\test\SLBPerfTestAutomation

• SLB Perf Tenantso ..\rd_cloudnet_n_slb\services\cloudnet\test\

SLBPerfTestAutomation\SLBPerfTenants

Thank you!!!