DISC IPL INED INNOVATION

TEST PERFORMANCE USING CLOUD

Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs)

Sayantam Dey (Architect, 3Pillar Labs)

OBJECTIVES

1. What are the benefits of using the cloud for testing performance over traditional means?

2. What are the various tools available for cloud based performance testing?

3. How can we use open source load testing frameworks in the cloud?

4. How to design a custom framework for rapid testing with large number of concurrent users?

TRADITIONAL PERFORMANCE TESTING

Load GenerationServer (High end

or multiple servers)

Application Server

Workstation

1. Deploy replica of the production environment in the LAN

2. Deploy a high-end server for generating load (on the same LAN)

3. Run agents of a load test tool like JMeter, Grinder or WAPT

4. Connect remotely to load generation server

5. Load generation server collects metrics and generates reports

CLOUD BASED PERFORMANCE TESTING

Load GenerationServer (High end

or multiple servers)

Application Server

Workstation

Cloud machinesin different geographies

Application Server

Workstation

BENEFITS OF USING THE CLOUD

Traditional Approach Cloud based Approach

Provisioning physical machines is costly, specially for medium sized companies

Large number of small, virtual machines running in bursts, ideal usage of cloud from a cost perspective

Manual provisioning of load generation server requires time and effort

Simpler provisioning, can even be automated using API

Application needs to be deployed in the same LAN

Application may be hosted at any public IP, also more realistic from latency point of view

Not dynamic to address any significant increase in the performance requirement

Distribute load generation over multiple instances which can be increased dynamically

Extremely difficult to simulate geographically distributed load

Cloud clusters in different geographic locations can simulate geographically distributed load

CLOUD APPROACH – CHOICES ?

Commercial Products

1 Record or upload scripts for usage scenarios using the web based interface

2 Configure testing parameters like number of virtual users, time for test, etc. through interface

3 Start the test through web based interface

4 Download reports and analyze

Open source frameworks

1 Create the required number of VM instances on the cloud, deploy the chosen load testing framework

2 Configure all the parameters for the framework and upload scripts written for the chosen framework

3 Run agents on the system under test and then start the test

4 Download reports and analyze

CLOUD APPROACH – CHOICES ?

Commercial Products

1 Record or upload scripts for usage scenarios using the web based interface provided by the vendor

2 Configure testing parameters like number of virtual users, time for test, % allocation to usage scenarios etc. through interface

3 Start the test through web based interface

4 Download reports and analyze

Open source frameworks

• Very popular, large user base• Very little programming

knowledge required• Handles many protocols• Plugin architecture for

extension

• Requires programming• Scripting in Jython or Clojure• Distributed framework

CLOUD APPROACH – CHOICES ?

Commercial Products

• JMeter compatible• Real time reports• 10K virtual users x 25 hours

@ $1000

Web Performance

• Selenium IDE scripts • Real browser or simple

request playback (virtual user)

• 10K virtual users x 25 hours @ $3750

• Custom scripts• Real time analytics• Custom pricing

Open source frameworks

Only few frameworks are mature enough to work over the cloud

Only the cost of cloud infrastructure, which is much less expensive

Completely flexible as scripting is done through open APIs

Can only be used through command line, user has to care about calculating VM requirement, creation and closure

Reports are basic and sometimes not helpful

A CUSTOM FRAMEWORK?

Commercial Products

Not many choices since there are only a few products in the market

May prove to be expensive for repeated load testing

May use recording or proprietary scripting, which may not be fully flexible

Provide web-based interface to directly run tests without worrying about virtual machines

Provide easy to use reports

X

Sizable investment - Suitable for organizations needing repeated performance tests

CASE STUDY - HAILSTORMConfiguration JMeter Amazon SUT

Provisioning Manager

JMeterScripts

Script Manager

JMeter Controller

Data Collector

Report Engine- JFreeChart

AMI- JMeter- JRE

Agent

System Under Test (SUT)

Agent

Agent

HTTP

11

CASE STUDY - HAILSTORM

Observations

Sharp rise in response time for over 50 users

“Detail Page” degrades significantly for 75 users and above

System is stable but peaks out at 75 concurrent users and starts degrading at 100 users

Maximum throughput of 3.2 transactions per second at 75 users is low

© Copyright 2012 3PILLAR GLOBAL All rights reserved

Sample Report Summary

12

CASE STUDY - HAILSTORM

© Copyright 2012 3PILLAR GLOBAL All rights reserved

Page Response time Avg. (ms)

Response time 90tile (ms)

% errors

Home Page 1384 2293 0.0

Login Page 559 764 0.0

Login 1050 1740 0.0

Browse 1996 2898 0.25

Category Browsing

2553 3601 0.0

Detail Page 5368 7231 0.27

Search 2054 2948 0.0

Sample Report for one iteration

CASE STUDY - HAILSTORM

• Application– A leading provider of online classroom and instruction

solutions

• Key Challenge– Required simulation of 40,000 concurrent users– Rapid and cost effective solution to capture load

metrics for large number of concurrent users

• The Solution– Different JMeter scripts/plans seamlessly integrated– Engaging reports pointing out bottlenecks generated