performance testing in the cloud

Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo

Performance Testing in the Cloud Stuart Moncrieff & Chris De Lorenzo

Iqnite Australia 2014

Presented at Iqnite Australia 2014 by Stuart Moncrieff & Chris De Lorenzo 2

www.myloadtest.com

About Us…

Stuart Moncrieff • Performance Testing consultant for over 10 years

• More than 10,000 people read something I have written about performance testing every month

− www.myloadtest.com

Chris De Lorenzo • Performance & Automation Test Lead at Tabcorp

• Lives and Breaths Performance Testing

http://www.myloadtest.com/


www.myloadtest.com

Presentation Roadmap

Part 1

• Tabcorp – Auto-scaling

• Vic Emergency – testing with a CDN

• Monitoring

• Auto-scaling

Case Studies & Examples

• Tabcorp – load testing at massive scale

• SA Health – using cloud-based load gens to test an internal app

Load Generators in the Cloud

• Testing outside the firewall

• When you need a lot of load generators quickly and cheaply

• Cloud costs

Load Testing Cloud-based applications

Case Studies & Examples

Part 2


www.myloadtest.com

Local

Load Gens Cloud-based

Load Gens

Cloud-based Test Envt

Local Test Envt

traditional load testing

Previous Cloud Experience

5 © Copyright 2014 Stuart Moncrieff, www.myloadtest.com

www.myloadtest.com

Part 1: Load Generators in the Cloud


www.myloadtest.com

Public Cloud (IaaS) Cost Calculation

Example costings for a 2-week test cycle • 50 Large Windows instances in the Sydney Region

• 10 days of test execution

• 4 hours of test execution per day

• 4 GB of data transferred per hour during testing

On-Demand Instance Type (in Sydney Region) Cost/Hour

Small (1.7GB, 1 CPU core/1 thread) $0.12

Medium (3.75GB, 1 CPU core/2 threads) $0.23

Large (7.5GB, 2 CPU cores/4 threads) $0.46

Extra Large (15GB, 4 CPU cores/8 threads) $0.92

Data transfer = $0.18/GB (charged for outbound data only)

Elastic IPs = $0.005/hour (when not in use)

Line item Calculation Cost

Instances 50 x 10 x 4 x $0.46 $920

Data 10 x 4 x 4 x $0.18

$29

IP Addresses 50 x 10 x (24-4) x $0.005 $50

TOTAL $999

A Performance Testing workload suits cloud-based infrastructure perfectly. Why have servers sitting idle for 20 hours/day?


www.myloadtest.com

Use Case 1: Testing outside of the firewall Your users are on the Internet, but your load generators are in the Data Centre. What potential bottlenecks are you not testing?

network devices load balancer

firewall system under test

end user

load generator

Data Centre

the Internet

Bottleneck:

Internet link

capacity

Bottleneck:

internal

firewall Bottleneck:

Network

devices


www.myloadtest.com

Why aren’t you testing from the cloud?

If you are testing a public-facing app, ask yourself these questions:

• What potential bottlenecks am I not testing?

• What additional risk is the project carrying due to this test approach?

• Is this the right person to be doing my performance testing?


www.myloadtest.com

Use Case 2: You need a lot of load generators It may be impractical to organise enough load generators for large tests

Virtual User Type Vusers/Load Gen

Web (HTTP/HTML) 2000

Flex 500

TruClient 50

RDP 20

Citrix ICA 20

SAP GUI 20

Note: these are approximate numbers only.


www.myloadtest.com

Case Study: Load testing an internal app at SA Health

Load testing an internal app using RDP virtual users. Impossible to get 50 load generators set up in time for test execution dates. • Set up load generators and RDS farm in AWS cloud.

• Main problem was connectivity from cloud to internal network of large government department.

• First time doing anything “cloud” for SA Health. Security team had no process for it. Escalated exemptions.

• Set up Amazon “Virtual Private Cloud”, and created a VPN connection to internal network. VPN was restricted to test environment IP addresses.

• Lots of meetings!

system under test The Cloud

LoadRunner Controller

load gens Remote Desktop servers

Data Centre

Install client application here

VPN


www.myloadtest.com

Case Study:

Melbourne Cup Race 7 Jumps @ 15:00

The race that stops a nation and hopefully not our site!

• Total transactions supported: 68,000,000

• Total bets: 11,000,000

• Total logins: 1,500,000

• Unique users per hour: 321,000

• Peak transaction rates per second

− Betting Peak: 3,046

− Paying Peak: 10,916

− Customer Transactions Peak: 1,317

− Mobile Native Apps Peak 3,846

$90.6 million in turnover on just the Melbourne Cup Race

Mobile Growth Trending Up

Large focus for performance testing

Melbourne Cup Year # of Bets From Mobile Native Apps

% Growth

2011 186,024

2012 837,489 350%

2013 1,281,022 53%

2014 Tuesday 4th November


www.myloadtest.com

Drivers to Cloud Infrastructure

Driver: Large revenue growth continuing on digital distribution channels • Cost

• Supports testing objectives

− Ease of use

− Elastic IP Capability

• Industry is Maturing

− Get On It! -> {Natural Selection}

− High Compute!

− API Integration

• Challenges

− Privacy | Terms and Conditions | Security | Accounts Payable

The driver to migrate load testing capability to the cloud?


www.myloadtest.com

Cloud Performance Testing - Tabcorp

Physical / Internal Cloud and External Cloud

Lesson’s learned from the TAB

• Still have internal physical and internal cloud machines

• Trial a number of products prior to defining final solution

• Amazon worked for us due to:

− Automation

• Small, Medium, Large Executions

− Firewall = No worries

− Quick Provisioning (min) / Cleanup (sec)

− Up and running in 1 week

− External location

• DDOS, Bandwidth, Data Cleansing services


www.myloadtest.com

Managing a large number of Load Generators What does it take?

Final Product • Final Product – Getting There…

− Invest in automation to take out the manual bottlenecks

− Execution Scheduled

• Bamboo and Ansible Tower

• All sized executions

− Status of execution is published

− Screens are available for everyone to see

− Large scale testing with minimal effort


www.myloadtest.com

So what does it look like?

This is just a sneak peak


www.myloadtest.com

Part 2: Load Testing Cloud-based apps


www.myloadtest.com

Infrastructure Monitoring

Good instrumentation is critical for performance testing and for your Operations team

Performance testing should also validate that the system has sufficient monitoring for future trouble-shooting

• Amazon CloudWatch vs. Azure Diagnostics

− Azure gives average of last hour

− CloudWatch counters are 1 minute averages (i.e. 1-minute granularity)

• CloudWatch console access is free, but you need to pay for API access

• Can write own values to counters. This is worth doing.

Maximise your visibility into system health and application behaviour


www.myloadtest.com

Hardware won’t save you!

Myth 1: Adding server capacity solves all performance problems

Myth 2: Infinite cloud capacity means that you don’t need to do any performance testing

• Bottlenecks could be due to code or configuration, not just capacity

• What are the operational costs of paying for infinite hardware?


www.myloadtest.com

“A backup that you haven't done a successful test recovery from is #notabackup.” -- Some guy on Twitter

“Don’t assume your auto-scaling works unless you have actually tested it under load.” -- Stuart Moncrieff


www.myloadtest.com

Auto-Scaling is not a Silver Bullet

If your system uses auto-scaling, you must test auto-scaling under load.

• What metric will trigger auto-scaling?

− Is it actually the metric that indicates a bottleneck?

• Optimise before auto-scaling.

− Tune until your workload is either CPU-bound, or I/O bound.

• How fast will auto-scaling kick in?

− Azure metrics based off last hour (moving average). A sudden spike in demand will not trigger auto-scaling.

− How long does it take an instance to spin up?

Auto-scaling never seems to work correctly the first time


www.myloadtest.com

War Story: Auto-scaling at Tabcorp

Considerations

• Even with a slow ramp-up solution failed due to 6 min provisioning time

− Based off CPU% > 50%

− Realized there are too many variables…

• Settled with setting “enough” capacity for “predictable peaks”

− Works for Tabcorp


www.myloadtest.com

Traps for new players

New (or important) test cases: • Can you redeploy while under load?

• Does alerting/monitoring work?

− Will your Ops team have enough visibility to solve problems in Production?

• Does Autoscaling work as expected?

• Failover/HA/DR test cases

− Between availability zones, between regions

− Do instances need to be “warmed up” before putting them under load?

• How will you test something behind a CDN?

− All traffic hits single POP of CDN, means less traffic hits backend (VicEmergency example)

Watch out for: • First time “cloud architects”

− You can always get help through your AM/SA

• How reproducible are your test results?

− “noisy neighbours”, CPU steal

− Inconsistent hardware for same instance type

• Routes for traffic from load gens

− All traffic routed over small pipe (MFB example)

• Restricting access to your test environment to load generators (using a whitelist on your firewall)

− Default limit of 5 static IP addresses (Elastic IPs)

• Monitoring in a PaaS environment

− You won’t have OS-level access to servers

It’s a trap!


www.myloadtest.com

Summary

Load generators in the Cloud • Cost model suits servers which are not needed 24

hours/day. Low-cost, pay as you go.

• Great for testing public-facing applications from outside the firewall.

• Useful for situations where you need a large number of load generators.

Load testing Cloud-based applications • Infrastructure monitoring is different. Invest the

time to set this up properly.

• Don’t assume auto-scaling will work perfectly without testing under load. New Cloud-specific test cases!


www.myloadtest.com

Thank you Please see us afterwards if you have questions

Email:

• [email protected]

Twitter:

• @StuartMoncrieff

LinkedIn

• http://au.linkedin.com/in/stuartmoncrieff

Website:

• http://www.myloadtest.com

Email:

• [email protected]

Twitter:

• @cjdelorenzo

LinkedIn

• http://au.linkedin.com/in/cjdelorenzo