qsic 2012 presentation automated testing of web

7/29/2019 QSIC 2012 Presentation automated testing of web

1/27

Developing a Feedback-DrivenAutomated Testing Tool for WebApplications

Scott McMaster and Xun Yuan (Google)

http://www.webtestingexplorer.org/


2/27

What Does a Modern WebApplication Look Like?

Multiple application tiers -- client and server. Frequently updated.

Especially the user interface.

Browser side: Hundreds/thousands of HTML elements. Dynamic -- always changing at runtime, frequently

asynchronously.


3/27

The Current State of WebApplication Testing

Heavy reliance on unit testing of individualclasses/modules. The majority of automated testing occurs here.

Ad-hoc and manual system/integrationtesting.

Inefficient automated testing through theuser interface: Hard to maintain due to rapid changes. Focused on observing the browser to the exclusion

of other application tiers.


4/27

The Future State:WebTestingExplorer

SupportsAutomated Exploratory Testing. Automated: Easy to replay (and regenerate)

interesting test cases.

Exploratory: Systematically discovers newapplication states. Testing: Includes test case generation, oracle

generation, real-time bug detection, and test casereplay.


5/27

WebTestingExplorer Advantages

Efficiently test through the user interface. Test like a real user. Don't miss integration issues and things that

are hard to observe. Behind the scenes in the browser. Or on other application tiers.


6/27

Design: Testability

Optimize for the case where testers havedetailed technical knowledge of theapplication-under-test.

And can improve testability (such as by addingunique id's).

Testable applications allow us to generatemore robust tests.


7/27

Design: Code-Based Configurationand Customization

Testers use Java code. Not a domain-specific language. Not declarative.

Not highly reliant on XPath expressions. Integrates with JUnit, WebDriver, Selenium,etc.


8/27

Design: Use Existing Libraries

Writing robust infrastructure to drive modernbrowsers and web applications is hard.

--> WebTestingExplorer is built on WebDriver. Inherits its low-level robustness. Easy for testers to extend.


9/27

Design: Performance, Stability, andEffectiveness

Many regions of a fully rendered webapplication are not interesting for testing. "Like" buttons, display ads, common banners, etc.

-->Easy to operate on subsets of the UI. Improves performance (less to look at). Improves stability (can skip unstable/changing

regions).

Improves effectiveness (can focus on the mostinteresting user scenarios).


10/27

WebTestingExplorer Workflow

Two Complementary Phases:

1. Exploratory Incrementally searches for new application states.

Writes out new test cases and oracles. Can search for bugs.

2. Replay Executes tests generated by the Exploratory phase.

For regression testing. A/B, comparison of actual and expected states.


11/27

Terminology

Element: An HTML element in the browser. Actionable Element: An element that the

application's end-user can interact with. Stateful Element: An element that the tester wants to

examine for state changes.

State: An abstraction of the current state of the webapplication, as defined by the tester.

Action: Performed on an element to cause something tohappen in the web application. Action Sequence: An ordered list of actions.

Test Case: An action sequence (deemed interesting onthe basis of state change) that is saved for future replay.


12/27

Workflow: Exploratory

Iteratively and systematically moving throughthe web application via the user interface. Monitors for different types of bugs.

Generates increasingly-long and sophisticated testcases (and oracles) for later replay.

Driven by feedback derived from state.


13/27

State: Definition

Def: An abstraction of the current state of theweb application, customizable by the tester.

WebTestingExplorer has a rich and

extensible notion of state. Can theoretically include a combination of:

Web browser state (HTML, DOM, cookies, ...) Web server state (session data, variables, ...)

Database state (persistent application data) User-defined (but some state

implementations are included and ready-to-use!)


14/27

State: Browser-Based Examples

Stateful HTML Elements All Count

Visible Selected subset (e.g. only 's)

Stateful Elements' Properties All (HTML+CSS) Selected subset (e.g. font-weight, color)


15/27

Exploration and State

Changes in observed state between actionsmove the exploration forward into longeraction sequences.

High-level algorithm:a. Pick an action sequence.b. Take a state snapshot.c. Do something.

d. Take another state snapshot.e. Compare a) and c).f. Different? Then:

Generate a new test case. Enqueue new action sequences to execute.

g. Repeat.


16/27

Exploration Algorithm

Execute IAS

Extend IAS

Extend AS

DEQUE

FS == SS

Write Test Case

Execute AS

Collect States

Create Oracles

No

Yes

AS: action sequenceAI: actionable elementIAS: initial action sequenceSS: start stateFS: final state


17/27

Exploration Algorithm

Execute IAS

Extend IAS

Extend AS

DEQUE

FS == SS

Write Test Case

Execute AS

Collect States

Create Oracles

, ,....,, ,....,..., ,....,

Yes

No


18/27

Oracles: Framework

WebTestingExplorer includes an oracleframework and some pre-defined oracles.

Can be used to detect bugs during

Exploratory and Replay phases. Oracles for regression testing can be

generated during Exploratory phase, usedduring Replay phase. May leverage initial and final state.


19/27

Oracles: Examples

HttpResponseOracle: Checks for allowedand disallowed HTTP response codes.

JSErrorCollectorOracle: Looks for Javascript

runtime errors. FinalStateCheckOracle: Uses initial and final

state for regression testing. Also:

The framework allows testers to plug in customoracle implementations.


20/27

Workflow: Replay Example(Regression Testing)

Exploration creates a test case. Test case includes saved final state for the

configured state checkers.

Replayer is configured with theFinalStateCheckOracle. Replayer replays the test case. At the end, a state snapshot is taken and

compared to the final state saved in the testcase.

If they are different, a failure is recorded.


21/27

Configuration

WebTestingExplorer is highly configurablefor both phases (Exploratory and Replay).

Can limit the state space and restrict the

action sequences to be explored. Examples:

Skip exploring common header and footer elementson the page.

Exclude ads from the Actionable and Statefulelements.

In an industrial web application, we have observedthat we can generate interesting test cases bychecking just two Stateful elements.


22/27

Configuration Points

Selectors for Actionable and Stateful web elements. Pre-packaged and custom Oracles. Action sequence filters to eliminate redundant or

uninteresting exploration paths.

Example: MaxRepeatedActionSequenceFiltereliminates action sequences that repeat the sameaction more than a set number of times.

Action generators to create custom actions on specific

elements. Action sequence prioritizerfor test case prioritizationduring exploration.

Custom wait conditions to increase stability whileexecuting an action sequence.


23/27

Example: Custom Action Generator

new TagActionGeneratorConfig("a") {

@Override

public Set generateActions(WebElementWithIdentifier elementWithId) {

Set actions = Sets.newHashSet();

String href = elementWithId.getElement().getAttribute("href");

if (href != null && href.startsWith("javascript:")) {

actions.add(new ClickAction(elementWithId.getIdentifier()));

}

return actions;

}

}

Only explore links that execute javascript:


24/27

Related Work

Browser automation frameworks (Selenium,WebDriver). Allow testers to automate test cases of their own

design. Model-based GUI test case generation

(GUITAR). Exploration model very focused on desktop

applications (menus and modal dialogs). Web site crawler (Crawljax). Diffs the browser DOM tree based on a similarity

metric. Defines invariants as Xpath expressions.


25/27

Future Work

Parallel exploration. Code coverage integration. Evaluate the effectiveness of different state

abstractions. Integrated state from multiple tiers of the

web application. Industrial applications and studies.


26/27

Questions?


27/27

Related Work: Crawljax

qsic 2012 presentation automated testing of web

Documents