amin milani fard: directed model inference for testing and analysis of web applications

Amin Milani Fard

Directed Model Inference for Testing and Analysis of Web

Applications

University of British ColumbiaOct 2015

http://www.knowdiff.net/

Modern Web Applications

Manual Testing and Maintenance

Tedious

Incomplete

Test Model Generation

Test Case Generation

Unit Test Fixture Generation

Code Maintenan

ce

P1. Test Model Generation• In model-based testing, models of program

behaviour are used to generate test cases.

• Dynamic analysis and exploration (crawling) derives test models for many automated testing techniques.

Most industrial web applications have a huge state-space.

Exhaustive crawling (BFS, DFS, or random search), can cause the state explosion problem.

Given a limited time, exhaustive crawlers can become mired in specific parts of the application, yielding poor coverage.

Covering the whole app is infeasible in a limited time, so …

RQ1. How can we derive test models for web applications more effectively compared to exhaustive exploration methods?

Amin Milani Fard, Ali Mesbah

Feedback-Directed Exploration of Web Applications to Derive Test

Models

24th IEEE International Symposium on Software Reliability Engineering (ISSRE),

2013

Test Model GenerationWe consider 4 properties for a test model:• Functionality Coverage: The amount of code coverage• Navigational Coverage: The amount of covering different

navigational branches • Page Structural Coverage: The amount of covering

heterogeneous DOM structures• Size: The number of edges in the SFG

How can we infer a model satisfying all these?

Feedback-directed Exploration• FeedEx uses the feedback obtained to predict

(1) which states should be expanded next(2) in which order events should be executed

• Repeat until a time/state limit• Take the crawler to state s with highest state score• Execute the fittest event on s based on event score

State score is a combination of• Code Coverage Impact: The amount of code coverage

increase• Path Diversity: The diversity (not sharing) of two paths • DOM Diversity: The diversity (dissimilarity) of two DOM

trees

Event Score• Event Productivity Ratio: Unexecuted events first.

Penalize events that result in an already discovered state, e.g. self-loops.

Evaluation and Results• Objects: 6 open-source JavaScript web apps• Fixed time: 5 minutes, no limitations on: depth, number

of states

Coverage0%

12%24%36%48%60%

DOM Diversity00000

Path Diversity00001

Test Model Size Test Suite Size0

150300450600750

DFS BFS RND FeedEx

10-28% imp 7-4000% imp 23-130% imp

38-86% imp 42-61% imp

DOM-based Testing

Crawling automates testing by exploring more states, but is limited in:• Proper input values• Choosing paths to explore• Generating effective assertions

RQ2. Can we utilize the knowledge in existing tests to generate new tests?

P2. Test Case Generation

Amin Milani Fard, Mehdi Mirzaaghaei, Ali Mesbah

Leveraging Existing Tests in Automated

Test Generation for Web Applications

29th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2014

Combining Manual and Automated Tests

- Input data and sequence- DOM elements to be asserted

- Automated crawling- Automated test case generation

Generated test cases

Testilizer idea

Extended State-Flow Graph

Initial State-Flow GraphHuman-

written test cases

Exploring Alternative Paths• Remain close to the manual-test paths

Initial State-Flow Graph

Extended State-Flow Graph

Regenerating Assertions(1) Reusing the same assertion

(2) Regenerating assertions for exact DOM element/region match

(3) Generating assertions for similar DOM region match

Assertion ReuseAn assertion on a shared state can be reused for a new test case.

Assertion RegenerationRepetition-based assertion regeneration

1. exact element-based assertions2. exact region-based assertions

Checked Element

Checked Element Region

Generating Similar Region Assertions

Inexact element/region repetition of a checked element/region can also be important for testing.

A classification problem: • Is a block level DOM element important to be checked by an

assertion?

<div id="header">

<div id="nav">

<div id="footer">

<div id="article"> <div id="sidebar

"><div id="section">

<div id="header">

<div id="nav">

<div id="footer">

<span id="main"> <span id="men

ue"><span id="content">

• 150% imp over the original test suite• 170% imp over the random assertion (RND) & exploration (RAND)• 37% imp over the random assertion (RND)

While code coverage was not our main goal• 30% imp over the original test suite• 18% imp over the random exploration

Fault Detection Rate0%7%

13%20%26%33%

ORIG RAND + RND EXND + RND Testilizer

Evaluation and Results

P3. Unit Test Fixture Generation

If an expected DOM element (test fixture) is not present, a JavaScript unit test may throw an exception or produce an incorrect result.

Test fixtures define states of the test environment before the test.

Proper DOM-based fixtures are required to achieve high coverage.

RQ3. How can we automate fixture generation for unit testing?

Challenges(1) DOM-related variables(2) Hierarchical DOM relations

Amin Milani Fard, Ali Mesbah, Eric Wohlstadter

Generating Fixtures for JavaScript Unit Testing

30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 2015

Apply the new fixture on DOMUnit Test

ConFix ideaInstrument the code Instrumented

CodeJavaScript Code

+Function under

test (FUT)

Solve constraints and generate a

fixture

Collect exec trace and deduce DOM-based constraints

Execute the FUT

Instrumentationtrace = [];function confixWrapper(statementType, statement, varList, varValueList, enclosingFunction, actualStatement) { trace.push({statementType: statementType, statement: statement, varList: varList, varValueList: varValueList, enclosingFunction: enclosingFunction, actualStatement: actualStatement}); return actualStatement;}function getConfixTrace() { return trace;}function dg(x) { return confixWrapper("return", "return confixWrapper(\"functionCall\", \"document.getElementById(x)\", [\"x\"], [x], \"dg\", document.getElementById(x));", [""], [], "dg", confixWrapper("functionCall", "document.getElementById(x)", ["x"], [x], "dg", document.getElementById(x)));}function sumTotalPrice() { sum = confixWrapper("infix", "sum = 0", [""], [], "sumTotalPrice", 0); itemList = confixWrapper("infix", "itemList = confixWrapper(\"functionCall\", \"dg('items')\", [\"items\"], ['items'], \"sumTotalPrice\", dg('items'))", [""], [], "sumTotalPrice", confixWrapper("functionCall", "dg('items')", ["items"], ['items'], "sumTotalPrice", dg('items'))); if (confixWrapper("condition", "itemList.children.length === 0", [""], [], "sumTotalPrice", itemList.children.length === 0)) confixWrapper("functionCall", "dg('message')", ["message"], ['message'], "sumTotalPrice", dg('message')).innerHTML = confixWrapper("infix", "confixWrapper(\"functionCall\", \"dg('message')\", [\"message\"], ['message'], \"sumTotalPrice\", dg('message')).innerHTML = \"Item list is empty!\"", [""], [], "sumTotalPrice", "Item list is empty!"); else { for (i = confixWrapper("infix", "i = 0", [""], [], "sumTotalPrice", 0); confixWrapper("loopCondition", "i < itemList.children.length", ["i", "itemList"], [i, itemList], "sumTotalPrice", i < itemList.children.length); i++) { p = confixWrapper("infix", "p = itemList.children[i].value", ["itemList.children[i]"], [itemList.children[i]], "sumTotalPrice", itemList.children[i].value); if (confixWrapper("condition", "p > 0", [""], [], "sumTotalPrice", p > 0)) sum += p; else confixWrapper("functionCall", "dg('message')", ["message"], ['message'], "sumTotalPrice", dg('message')).innerHTML += " Wrong value for item " + i; } confixWrapper("functionCall", "dg('total')", ["total"], ['total'], "sumTotalPrice", dg('total')).value = confixWrapper("infix", "confixWrapper(\"functionCall\", \"dg('total')\", [\"total\"], ['total'], \"sumTotalPrice\", dg('total')).value = sum", [""], [], "sumTotalPrice", sum); }

(1) Extract DOM-based Constraints

(2) Transform them into XPath

(3) Solve it using an available XML XPath solver

Evaluation and Result

• Up to 67% imp in statement coverage

• Up to 300% imp in branch coverage

P4. Code Maintenance• JavaScript is challenging to

maintain.

• Code smells adversely influence program comprehension and maintainability.

• Code smells detection is time consuming.RQ4. Which JavaScript code smells are prevalent in

practice and how can we support automated code refactoring?

Amin Milani Fard, Ali Mesbah

JSNose: Detecting JavaScript Code Smells

13th IEEE International Conference on Source Code Analysis and Manipulation (SCAM), 2013

JavaScript Code Smell DetectionStatic and dynamic code analysis is used to monitor and infer information about objects, functions, and code blocks.

Evaluation and Results

• Evaluated 11 JavaScript/Ajax web applications.

• Effective code smells detection (93% precision, 98% recall)

• Lazy object, long method/function, closure smells, JS/HTML/CSS coupling, and excessive global variables, are the most prevalent smells.

• Strong, and significant +correlation between {LOC, # functions, # JS files, and CC} and the types of smells, and weaker correlation with the # smell instances.

Some topics to work on• Applying learned tests on similar applications

• Generating unit tests based on client-side code similarity and DOM-based tests based on DOM sate similarity

• JavaScript code refactoring• Suggesting and applying refactoring for code

smells

• Understanding test failures and root causes

• Generating integrated tests using existing unit tests

Test Model Generation

Test Case Generation

Unit Test Fixture Generation

Code Maintenan

ce