1

Load Testing Ajax Apps

using head-less browser tools

NoVaTAIG

April 13, 2011

Gopal Addada and Frank Hurley

Cigital Inc.

Agenda

About Cigital

Background : AJAX and Load Test requirements

Tools research

HTMLUnit

Virtual User (VU) Generation

Results and reporting

Pros and Cons

Limitations and Future work

3Dec10

Software Assurance services

Software Quality

Agile testing

Test automation

Continuous integration

Test process improvement

Software Security

Secure design

Secure coding

Security testing

Continuous integration

Background: AJAX

Web 2.0 !

Interactive

Several AJAX frameworks are being used

GWT, jQuery, MochiKit, Adobe Flex, Dojo, YUI

COTS tools (claim to) support Ajax load tests:

LoadTester, Neotys, LoadRunner(HP)

Very expensive (per license per Virtual User)

Open source solutions

Jmeter, Grinder and openSTA

Taken from Aztecsoft Whitepaper [1]

Background: AJAX

Please refer to Descriptive Picture at URL [2]:

http://www.adaptivepath.com/uploads/archive/images/publication

s/essays/ajax-fig2_small.png

Classic Web applications AJAXy web applications

Requests sent as direct response to

user actions

Requests can happen due to user

actions or asynchronously at intervals

determined by AJAX engine

Requests are normally <form> that

use GET/POST methods

Can be of any form supported by

remote application server (e.g.

XMLHttpRequest)

Browser - server involves requests for

complete pages

Typical requests involve updating

parts of already loaded page

The response from server involves

entire new page which is rendered at

client browser

The response is intended for small

portion of page which is interpreted by

AJAX engine on client side and

updates DOM of current page

Partially from Aztecsoft Whitepaper [1]

Background: AJAX

Load Test Tool Requirements for Web

Need to simulate User behavior using Virtual

Browsers

Real browsers don’t work:

Do not scale well as running multiple browser tests from

test machines is not feasible

Intent is to test the Server performance

JMeter and Grinder are prominent open-source tools

Use HTTPClient implementations and are multi-threaded

GUI to create load tests

Do not support Java Script and hence can’t simulate AJAX

engine behavior

Classic Web applications AJAXy Web applications

Need any virtual (headless) browser that

simulate user actions using HTTP client

Need any virtual (headless) browser

that simulate user actions using “HTTP

client with JavaScript Support”

Involves testing mostly server, client

behavior may not have to be replicated

(e.g. cross-browser does not matter).

Simulating concurrent HTTP GETs/POSTs

is enough (e.g. Jmeter, Grinder)

Browser runs AJAX Engine (JS calls) ->

need to simulate client behavior to

maximum extent possible

Response has to be asserted and

response times may need to be captured.

Server has to be monitored for CPU

performance, memory usage and DB

performance

Response has to be asserted and

response times may need to be

captured. Server has to be monitored

for CPU performance, memory usage

and DB performance

Load Test Tool Requirements for Web

Jmeter

One of the best load test tools, but does not support JS calls

HttpUnit

Good Headless browser tool and No JS support

Grinder – Good tool, but no known JS support

HtmlUnit

100% Java-based headless browser

Supports HTTP Requests, HTML Parsing, JS execution, CSS

Easy to script tests (similar to Selenium)

WebTest

Built on HtmlUnit and tests written in Groovy or XML

Can potentially be used for load test design

Easier and non-programmer friendly

Tool research (open-source)

@Test

public void testGoogle() throws Exception {

WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3);

HtmlPage gPage = webClient.getPage(“http://www.google.com”);

assertEquals(“Google", gPage.getTitleText());

HtmlUnit qTxtbox = (HtmlInput)gPage.getElementByName(“q”).get(0);

qTxtbox.setValueAttribute(“NoVaTAIG”);

HtmlElement btn = gPage.getFirstByXPath(“//input[@value=\” I'm Feeling Lucky\”]”);

HtmlPage gPage2 = btn.click();

asserEquals(“Northern Virginia Test Automation Interest Group”,gPage2.getTitleText());

}

}

HtmlUnit

HtmlUnit (contd.)

SHOW ARCHITECTURE DIAGRAM from MARC’s

PRESENTATION

HtmlUnit (contd.)

AJAX Libraries Supported

CurvyCorners

Dojo

ExtJS

GWT

JQuery

MochiKit

Prototype

Sarissa

Yahoo UI

...

Approach for our load tests

Complete Black-box approach simulating a browser

(Vuser) and perform user actions

Write functional tests for pages to be load tested

Generate desired number of threads

Each thread calls functional test(s)

Each thread can use different user

To simulate a close-to-reality scenario

Each thread may have to simulate behavior of

different browser (IE, Firefox, Chrome)

Virtual users (VU)

ThreadPoolMain

T

Thread 1

TThread 3

T

Thread 2

T

Thread n

T

HtmlUnit

CodeT

Test Status Data reported back to main

HtmlUnit

CodeT

HtmlUnit

CodeT

HtmlUnit

CodeT

Approach

//Every thread calls this function which has sequence of tests

public void runGTest() throws Exception{

HtmlPage currentPage;

webClient.setUseInsecureSSL(true);

final HtmlPage loginHomePage = login(url,username,passwd);

WebAssert.assertTextPresent(loginHomePage, "Change security

question");

currentPage = loginHomePage;

goToGmail(loginHomePage); /** INVOLVES AJAX CALLS**/

logout(currentPage);

}

HtmlUnit code demo

//Sample test involving Ajax calls

private HtmlPage goToGmail(HtmlPage dashboardPage) throws Exception{

webClient.setAjaxController(new NicelyResynchronizingAjaxController());

final HtmlAnchor gmailLink = dashboardPage.getFirstAnchorByText("Gmail");

final Page gmailPage = gmailLink.click(); // AJAX Engine call (JS)

WebAssert.assertElementPresent((HtmlPage)gmailPage,"loading");

WebAssert.assertLinkPresentWithText((HtmlPage)gmailPage,"Load basic”);

WebAssert.assertTitleContains((HtmlPage)gmailPage, "Gmail");

return dashboardPage;

}

HtmlUnit code demo (Contd..)

Thread class

public class GmailThread implements Runnable {

private int userIndex = -1;

public GmailThread(int userIndex) {

this.userIndex = userIndex;

}

public void run() {

String origName = Thread.currentThread().getName();

Thread.currentThread().setName("user_" + userIndex);

GAccountAccess gTest= new GAccountAccess();

gTest.runGTest();

}

}

Test Thread Details class (New Object Passed

from main to all the way to HtmlTest)public class TestStatusData {

private long threadStartTime;

private long threadEndTime;

private String tUsername;

private String tPasswd; // Don’t save password – just place holder

Private threadPage1ResponseTime;

Private threadPage2ResponseTime;

Private threadPage3ResponseTime;

private long threadLoginTime;

private long threadLogoutTime;

String threadCurrentStatus;

String tComment;

//sample setter and getter

public void setSFTPLoginTime(long sftpLoginTime){

threadSFTPLoginTime = sftpLoginTime;

}

public long getSFTPLoginTime(){

return threadSFTPLoginTime;

}

}

Thread Pool (Main class) – *** Can be used as

black box for most load tests. Details not really important.

Partial code (Will be posted in blog)private ExecutorService userThreadPool = null;

private HashMap<Future<GmailThread>, GmailThread> taskMap =

new HashMap<Future<GmailThread>, GmailThread>();

userThreadPool = Executors.newFixedThreadPool(concurrentThreadCount, new

NamedThreadFactory("UserPool"));

for (int i = 0; i < totalRuns; i++) {

testData[i] = new TestStatusData(); // Detailed in previous slide

int userIndex = i % usernames.length; //randomize username/paswords to be

used

testData[i].setTUsername(usernames[userIndex]);

TestFlexorAuthThread handler = new

TestFlexorAuthThread(testData[i],usernames[userIndex],passwds[userIndex]);

Future<TestFlexorAuthThread> serverTask = userThreadPool.submit(handler,

handler);

taskMap.put(serverTask, handler);

}

awaitTermination();

userThreadPool.shutdown();

Serves the purpose without COTS tools

Complete control and Minimal Java needed

Can be cronned for run overnight builds

Underlying HtmlUnit continuously updated and supports

new frameworks

Can simulate multiple Browser in single Load test

HtmlUnit Simulates IE7, IE8, Firefox 2, Firefox3

Can configure each thread for different browser

Generic load generation

Load generation code can be re-used for other web apps

Used for client/server applications (SFTP, FTP, SMTP)

Good

Does not have reporting UI Working on incorporating into JMeter

Investigating incorporating into Grinder

HtmlUnit does not support all AJAX libraries (RIAs)

Adobe Flash, Silverlight, WPF

Scalability: Limited by JVM and CPU availability Adding additional hardware helps, or making tests distributed

Hard to maintain as UI/ object properties change It’s an issue with every load/functional test tool

Not very friendly for non-programmer Very basic programming needed

Use Canoo WebTest instead of HtmlUnit

We are currently working on this

Bad

JVM tuning

sponse haUse as powerful machine as possible (multi core)

JVM assigned to max possible (32-bit, 64-bit)

All the HTMLUnit tests have to be thread-safe

To be incorporated into load generation framework

Fine-grained Logging needed for trouble shooting

E.g. To know what response server is sending back?

Every possible exception better be handled in HtmlUnit

code and Thread code for failure analysis

Lessons Learned

Using TestNG ThreadPool framework for HtmlUnit tests

WebTest + “Groovy multi-threading”

Jmeter Groovy Sampler+ WebTest (needs extra effort)

Other ways to perform load tests

Use Groovy with WebTest Canoo to make tests non-programmer friendly

Work on potential integration with Jmeter

Isolate Load generation and load execution (LoadRunner/Grinder approach of Controller + VUgen)

For large-scale tests Additional Virtual Machine support

Cloud based support

Develop distributed version of tool for CLOUD

IN Progress! Later : Develop WebUI for configuration purposes

Future Work

Questions, Comments and

Suggestions?

27

Gopal’s Load testing Blog:

http://webloadtest.blogspot.com

Please post comments and suggestions

Updates on load test framework(s) will be

posted

References

[1]

http://www.crn.in/Resources/Docs/Aztecsoft_Whitepaper_Perf

ormance_Testing_AJAX-based_Applications.pdf

[2] http://www.adaptivepath.com/ideas/e000385

28