scaling regression testing to large software systems · scaling regression testing to large...
TRANSCRIPT
Scaling Regression Testing toLarge Software Systems
Alessandro OrsoCo-authors: Nanjuan Shi,
Mary Jean Harrold
College of ComputingGeorgia Institute of Technology
Supported in part by National Science Foundation (awards CCR-0306372, CCR-0205422, CCR-9988294, CCR-0209322, SBE-0123532, and CCR-0080900) and Boeing Commercial Airplane Group.
FSE 04
Testing of Evolving Software
P’Program
P
Test SuiteT
T
Trerun
T - Trerun
T’
P’’
T’rerun
T’ - T’rerun
…
Time to rerun T
time
Analysis Time Time to rerun Trerun Savings
FSE 04
Previous and Related Work
Ideal solution: two-phase approach Class-Level analysis subset of P Stmt-Level analysis on the subset Trerun
P
T(coverage info)
P’Trerun
Retest
Stmt-LevelAnalysis
Time to rerun T
time
Analysis Time Time to rerun Trerun Savings
[OOPSLA01]
FSE 04
Previous and Related Work
Ideal solution: two-phase approach Class-Level analysis subset of P Stmt-Level analysis on the subset Trerun
P
T(coverage info)
P’Trerun
Retest
Stmt-LevelAnalysis
Time to rerun T
time
Analysis time Time to rerun Trerun
Precise, Expensive Techniques
Low-LevelAnalysis
Jboss, web application server, 1 million LOC
FSE 04
Previous and Related Work
Time to rerun T
time
Analysis time Time to rerun Trerun
P
T(coverage info)
P’Trerun
Efficient, Imprecise Techniques
High-LevelAnalysis
FSE 04
Previous and Related Work
Time to rerun T
time
Analysis time Time to rerun Trerun
P
T(coverage info)
P’Trerun
Efficient, Imprecise Techniques
High-LevelAnalysis
Related Work Efficient, less precise techniques
White and Leung [CSM92] Chen, Rosenblum, and Vo [ICSE94] Hsia et al. [SMRP97] White and Abdullah [QW97] Ren et al. [OOPSLA04]
Expensive, more precise techniques Binkley [TSE97] Rothermel and Harrold [TOSEM97] Vokolos and Frankl [RQSSIS97] Ball [ISSTA’98] Rothermel, Harrold, and Dedhia [JSTVR00] Harrold et al. [OOPSLA01] Bible, Rothermel, and Rosenblum [TOSEM01]
FSE 04
Proposed Solution
Two-phase approach Class-Level analysis subset of P and P’ Stmt-Level analysis on the subset Trerun
P
T(coverage info)
P’Trerun
Two-Phase Technique
Class-LevelAnalysis
Stmt-LevelAnalysis
Subsetof P
Subsetof P’
FSE 04
Outline
Background Technique Empirical Evaluation Conclusion
FSE 04
Motivating Example
class A { void foo() {…} }class B extends A {
}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
P
FSE 04
Motivating Example
class A { void foo() {…} }class B extends A {
}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
P P’ class A { void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Motivating Example
class A { void foo() {…} }class B extends A {
}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
P P’ class A { void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Motivating Example
class A { void foo() {…} }class B extends A {
}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
P P’ class A { void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Class-Level Analysis
class B extends A {
}
P P’
class B extends A { void foo() {… }}
FSE 04
Class-Level Analysis
class A { void foo() {…} }class B extends A {
}
class C extends B {}
P P’ class A { void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
FSE 04
Class-Level Analysis
class A { void foo() {…} }class B extends A {
}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
P P’ class A { void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
FSE 04
Class-Level AnalysisP / P’
A
D
F
B
E
C
Inheritance edgeUse edge
Interclass Relation Graph(for P an P’)class A {
void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Class-Level AnalysisP / P’
A
D
F
B
E
C
Inheritance edgeUse edge
Interclass Relation Graph(for P an P’)class A {
void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Class-Level AnalysisP / P’
A
D
F
B
E
C
Inheritance edgeUse edge
Interclass Relation Graph(for P an P’)class A {
void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Class-Level AnalysisP / P’
A
D
F
B
E
C
Inheritance edgeUse edge
Interclass Relation Graph(for P an P’)class A {
void foo() {…} }class B extends A { void foo() {… }}
class C extends B {}
class D {void bar() { A ref=null; switch(somevar) { case ‘1’: ref=new A(); break; case ‘2’: ref=new B(); break; case ‘3’: ref=new C(); break; } ref.foo();} }
class E extends D {}class F { void bar(D d) {…} }
FSE 04
Example: Stmt-Level Analysis
ref.foo()
A.foo()
…
C
B
A…
…
A.foo()
…
A
C
Bref.foo()
…
…B.foo()
…
A.foo()
B.foo()DangerousEdge
G’ (excerpt)G (excerpt)
class Aclass B {…}class Cclass D { void bar() {…; ref.foo(); …} } }
Subset of P’Subset of P class Aclass B {… void foo() {…} … }class Cclass D { void bar() {…; ref.foo(); …} } }
FSE 04
Example: Stmt-Level Analysis
ref.foo()
A.foo()
…
C
B
A…
…
A.foo()
…
A
C
Bref.foo()
…
…B.foo()
…
A.foo()
B.foo()DangerousEdge
G’ (excerpt)G (excerpt)
class Aclass B {…}class Cclass D { void bar() {…; ref.foo(); …} } }
Subset of P’Subset of P class Aclass B {… void foo() {…} … }class Cclass D { void bar() {…; ref.foo(); …} } }
Test cases to be rerun:Test cases in T thatexecute the call nodewith ref’s dynamic typebeing B or C
FSE 04
Outline
Background Technique Empirical Evaluation Conclusion
FSE 04
Empirical Setup
ToolDejaVOO
Subjects
Three empirical studies1. Effectiveness of Phase 12. Precision gain of Phase 23. Overall savings in retesting time
639
200
707
TestCases
32 min1,0002,4035Jboss
74 min1678245Daikon
54 min705255Jaba
RetestTimeKLOCClassesVersionsProgram
FSE 04
RQ1: Effectiveness of Phase 1
P
T(coverage info)
P’Trerun
TwoPhase
Class-LevelAnalysis
Subsetof P
Subsetof P’
Stmt-LevelAnalysis
FSE 04
RQ1: Effectiveness of Phase 1
Average: 37%
Time < 14 seconds
FSE 04
RQ3: Overall Savings in retesting Time
Time to rerun T
time
Analysis time Time to rerun Trerun Savings
FSE 04
RQ3: Overall Savings in Retesting TimeRerunAll TwoPhase
FSE 04
RQ3: Overall Savings in Retesting TimeRerunAll TwoPhase
Savings in RegressionTesting Time:
TwoPhase vs. RerunAll
Jaba:19%Daikon:36%Jboss: 63%
FSE 04
Outline
Background Technique Empirical Evaluation Conclusion
FSE 04
Contribution
Extended our existing regression-testing technique Considerable increase in efficiency Same precision
First RTS technique that, at the same time: is safe combines coarse and fine-grained analysis handles the features of the Java language
Tool (DejaVOO) that implements the technique Empirical results showing the effectiveness of the
technique
FSE 04
Current and Future Work
Further investigate cost-effectiveness ofthe technique Continuous integration Larger systems
Investigate test-suite augmentation Based on changes (program differencing) Based on field-data (impact analysis)
FSE 04
Questions?