x-pert: accurate identification of cross-browser issues in web applications

X-PERT: Accurate Identification

of Cross-Browser Issues in Web Applications

Shauvik Roy Choudhary, Mukul Prasad, Alessandro Orso

Labs of America

Partially supported by

Web Applications

2

4

Mozilla Firefox

Internet Explorer

5

Mozilla Firefox

Internet Explorer

6

Google Chrome

7

Apple Safari

8

Apple Safari Mozilla Firefox

9

Mozilla Firefox

10

Mozilla Firefox

Internet Explorer

11

Mozilla Firefox

Internet Explorer

?

?

Web Developer

1211Browser versions last

year

Source: StatCounter.com

56Browser versions last

yearwith at least 1% market share

150mn visits/month (StatCounter.com)

Challenges

Manual inspectionis expensive

DOM differs between browsers

Mimic end user’sperception

Produce readable, effective & actionable

reports

EngineeringIssues…

Modern apps havemany dynamic screens

Detection ofCross-Browser Issues

(XBI)

Web Application

Model Generation

Model Comparison

Error Report

δ

δ

Model Definition and Comparison

WebDiff [ICSM’10]Roy Choudhary and Orso

CrossCheck [ICST’12]Roy Choudhary, Prasad and Orso

CrossT [ICSE’11]Mesbah and Prasad

Screen Transition Graph

Screen Model

Screen image + geometry

DOM Tree

Study of Real World Cross-Browser Issues

(XBIs)• Examined 100 websites• Found 23 with XBIs

• XBI TypesoBehavioral XBIs

oContent XBIs

oStructural XBIs

9% 22

%30%

57%

TextVisual

Random URL Generator

Limitations of existing techniques

• Miss Structural XBIs

• Focus on symptoms rather than causeso Duplicate reportso Low Precision

ComprehensiveXBI Detection

• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual

Example

Example

Behavior XBI9%

ExampleContent

XBI(Visual) –

30%

Content XBI

(Text) – 22%

Structural XBI57%


• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual


• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual

Behavior check using Graph Isomorphism

= ?CrossCheck [ICST’12]Roy Choudhary, Prasad and Orso



• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual

Text Contentcheck using String equality

= ?





• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual

Visual Contentcheck using Image Matching

= ?Computer

Vision



Detecting Structural XBIs

• Intuition: Disruption in the relative arrangement of web page elements leads to Structural XBIs

• Idea: Abstraction to represent relative arrangement and compare it to expose errors

Alignment Graph (AG)

• A graph capturing structural relationships between elements (rectangles) on the webpage

• An AG has:oContainment relationshipsoAlignment relationships oAttributes

aboveright-alignleft-align

Step 1: AG Construction

(x1, y1) (x2,

y2)

1

2

3

4

56

7

8

9

10

11

12

13

14

15

16

17

18

Step 1: AG Constructiona. Containment Relationship

e contains e’ iff• ( x1 ≤ x1’ ∧ y1 ≤ y1’ ∧ x2 ≥ x2’ ∧ y2 ≥ y2’ ) , and• if same bounds and XPath(e) ≤ XPath(e’)

(x1, y1)

(x2, y2)

(x1’, y1’)

(x2’, y2’)

e

e’

Step 1: AG Constructiona. Containment Relationship

Step 1: AG Constructionb. Alignment Relationship

Step 1: AG Constructionc. Attributes

Button 1 Button 2

What alignment attributes can we infer?

• Button 1 is on the left of Button 2

• Button 1 and Button 2 have their tops aligned

• Button 1 and Button 2 have their bottoms aligned

leftOftop-alignbottom-align

y1 = y1’

y2 = y2’

x2 < x1’

Step 1: AG Constructionc. Attributes

Button 1 Button 2

leftOftop-alignbottom-align

Step 2: Matching AGs

TextVisual


• Behavioral XBIs

• Content XBIs

• Structural XBIs

TextVisual

ReportGeneratio

n

Model Generatio

n

BehaviorChecker

ElementMatcher

Structure

Checker

ContentChecker

Model Comparison

✔ ✔

✔

Empirical Evaluation

Tool: X-PERT (Cross-Platform Error ReporTer)

Research Questions:• RQ1: Can X-PERT find XBIs in real web

applications?

• RQ2: How does X-PERT compare to the state-of-art?

Experimental Protocol

v14.0.1 v9.0.9v/s

Ran X-PERT on Subject Applications

Subjects

+ = XBIs(Ground Truth)Manual Check

Error Reports Manual Check

=False Positives & Negatives+

Subjects

Name Type States TransitionsDOM Nodes (per screen)

max min average Organizer Productivity 13 99 27482 10001 13051 GrantaBooks Publisher 9 8 37800 15625 25852 DesignTrust Business 10 20 26437 7772 18694 DivineLife Spiritual 10 9 140611 9082 49886 SaiBaba Religious 13 20 42606 524 12162 Breakaway Sport 19 18 45148 8191 13059 Conference Information 3 12 878 817 853 Fisherman Restaurant 15 17 39146 15720 21336 Valleyforge Lodge 4 12 5416 4733 5046 UniMelb University 9 8 15142 12131 13792 Konqueror Software 5 4 17586 15468 16187 UBC Club 7 7 20610 7834 12094 BMVBS Ministry 5 20 19490 12544 15695 StarWars Movie 10 9 28452 19719 22626

800 - 140K elements

Prior Art (6)

Example and Survey (4)

Random (4)

Effectiveness

NAME XBIBEHAVIOR STRUCTURE

CONTENTTOTALTEXT IMAGE

TP FP TP FP TP FP TP FP TP FPOrganizer 10 1 0 9 0 0 0 0 0 10 0

GrantaBooks 27 16 0 11 0 0 0 0 0 27 0

DesignTrust 7 2 0 5 3 0 0 0 0 7 3DivineLife 11 7 0 3 6 1 0 0 0 11 6SaiBaba 5 2 0 2 9 0 0 0 0 4 9Breakaway 13 0 0 10 2 0 0 0 0 10 2Conference 7 2 0 3 0 1 0 1 0 7 0Fisherman 5 1 0 3 1 0 1 1 0 5 2Valleyforge 3 0 0 2 2 0 0 1 0 3 2UniMelb 2 2 0 0 0 0 0 0 1 2 1Konqueror 0 0 0 0 0 0 0 0 6 0 6UBC 0 0 0 0 0 0 0 0 0 0 0BMVBS 1 0 0 0 0 0 0 0 0 0 0StarWars 12 0 0 12 0 0 0 0 0 12 0TOTAL 103 33 0 60 23 2 1 3 7 98 31

RQ1: Can X-PERT find XBIs in real web applications?

Answer: Yes

Improvement

NAME XBIX-PERT CROSSCHECK

Prec FP Recl Dp Prec FP Recl DpOrganizer 10 100% 0 100% 0 80% 2 80% 13GrantaBooks 27 100% 0 100% 0 96% 1 100% 0DesignTrust 7 70% 3 100% 0 5% 122 86% 3DivineLife 11 65% 6 100% 0 29% 24 91% 3SaiBaba 5 31% 9 80% 0 7% 53 80% 10Breakaway 13 83% 2 77% 1 13% 49 54% 12Conference 7 100% 0 100% 0 100% 0 100% 0Fisherman 5 71% 2 100% 0 44% 5 80% 8Valleyforge 3 60% 2 100% 0 50% 1 33% 0UniMelb 2 67% 1 100% 0 7% 27 100% 0Konqueror 0 NA 6 NA 0 0% 11 NA 0UBC 0 NA 0 NA 0 0% 1 NA 0BMVBS 1 NA 0 NA 0 0% 2 0% 0StarWars 12 100% 0 100% 0 10% 91 83% 3

TOTAL 103 77%31 95% 1 32% 389 74% 52

X-PERT

Precision = 77% (45%)

Recall = 95% (14%)

RQ2: How does X-PERT compare to the state-of-art?

Answer: X-PERT has better precision and recall

Experimental Data & Tool

• Releaseo Experiments (Crawl Data + Reports)o Layout testing algorithm

implementation

http://gatech.github.io/xpert

http://gatech.github.io/xpert

51

Related Worko Industrial tools

• Test Suites for Web Browsers – Acid and test262

• MS Expression Web, BrowserShots, BrowserStack

• Browsera, MogoTest, BrowserBite

o Research techniques

• Eaton & Memon [IJWET’07]

• Dallmeier, Burger, Orth & Zeller [JSTools’12]

Contributions

Empirical evaluationshows effectiveness

New Structural XBIDetection Algorithm

Comprehensive Technique to detect XBIs

Study of Real World XBIs

Thank YouShauvik Roy Choudhary, Mukul Prasad, Alex Orso

x-pert: accurate identification of cross-browser issues in web applications

Documents