computer science automated test data generation for aspect-oriented programs mark harman (king’s...
TRANSCRIPT
Computer Science
Automated Test Data Generation for Aspect-Oriented Programs
Mark Harman (King’s College London, UK)
Fayezin Islam (T-Zero Processing Services, US)
Tao Xie (North Carolina State University, US)
Stefan Wappler (Berner & Mattner, Germany)
Computer Science
Background
Automated testing of aspect-oriented programs
• Testing aspectual composition behavior (pointcut behavior) [Ferrari et al. ICST 08, Anbalagan&Xie ISSRE 08, …]
• Testing aspectual behavior (advice behavior) [Xie&Zhao AOSD 06, Xie et al. ISSRE 06, ...]
Computer Science
Testing Aspectual Behavior
• Aspect weaving (e.g., with ajc, abc)– Aspect Class (bytecode)
– Advice Method (bytecode)
• Straightforward unit testing: feed aspect classes to OO test generation tools based on bytecode– Issues: arguments can be thisJoinPoint or AroundClosure objects
• Aspectra: generate tests for woven classes but focus on aspectual behavior– Feed woven classes to OO test generation tools
– Base classes == “test drivers”
• Leverage existing OO tools for testing AOP programs
[Xie&Zhao AOSD 05]
E.g., Parasoft Jtest (random testing tool)
Computer Science
ExampleProgram
Under Test
+
Computer Science
New Contributions
• A new system of automated test data generation for AOP based on Search-Based Testing, i.e., evolutionary testing– Input domain: receiver object, method parameters
– E.g., account.debit(amount)
• Empirical studies to demonstrate the benefits of the system in AOP structural testing– Effectiveness: better than random testing
– Efficiency: AOP domain reduction techniques
– Efficiency: focusing test effort on aspectual branches to improve efficiency
Computer Science
What is Search-Based Testing?
In search-based testing, we apply search techniques to search large input spaces, guided by a fitness function.
Fitness function measures how good/close one input is in reaching the goal e.g., covering true branch of
Computer Science
Evolutionary Algorithms
Selection
Insertion
Recombination
Mutation
Fitness evaluation
End?
chromosome
Computer Science
Evolutionary Testing
Selection
Insertion
Mutation
End?
Test cases
Recombination
Fitness evaluation
Computer Science
Evolutionary Testing
Selection
Insertion
Mutation
End?
Test cases
Execution
Recombination
Fitness evaluation
Computer Science
Evolutionary Testing
Selection
Insertion
Mutation
End?
Test cases
Execution
MonitoringRecombination
Fitness evaluation
chromosome
Method seq
Computer Science
Structural Evolutionary TestingStructural Evolutionary Testing
Target:true branch
Computer Science
Fitness = (A - B) + 1
101 = (100 – 0) + 1
51 = (100 – 50) + 1
0 = (100 – 101) + 1
Evaluation of predicate in a branching condition
if (A < B)
Structural Evolutionary TestingStructural Evolutionary Testing
Lower fitness value, the better 0 fitness value, reach the target
Target:true branch
Computer Science
AOP Domain Reduction - Motivation
• Input domain: receiver object, method parameters– E.g., account.debit(amount)
• Not all input variables are relevant to the coverage of the target branch inside aspects
Target:false branch
…
Computer Science
Slicing-based Domain Reduction
• Irrelevant-input-variable identification• Start with slicing criterion: predicates of target branches
• Extract backward program slices (based on data and control dependence)
• Identify relevant input variables: input variables showing up in the slices: account.debit(amount)
• Domain reduction: searching for only relevant input vars
Target:false branch…
Computer Science
EvolutionaryAspectTester (EAT) System Implementation Indus slicer
[Ranganath et al. 07]
EvoUnit [Wappler 08]
Aspectra [Xie&Zhao 06]
Computer Science
Evaluation Benchmarks
14 benchmarks from [Xie&Zhao 06, Rinard et al. 04, Dufour et al. 04, Hannemann&Kiczales 02]
Computer Science
Study 1: Assessment of evo testing
RQ 1.1. Can evolutionary testing outperform random testing for AOP testing?
Computer Science
RQ 1.1: Assessment of evo testing
Coverage improvement of evolutionary testing over random testing
Better branch coverage on 5/14 benchmarks
43%
Computer Science
RQ 1.1: Assessment of evo testing cont.
61%
Effort reduction of evolutionary testing over random testing
Effort reduction on 9/14 benchmarks
Computer Science
Findings: Assessment of evo testing
RQ 1.1. Can evolutionary testing outperform random testing for testing aspect-oriented programs?
•Better branch coverage on 5/14 benchmarks (0%~43%)•Effort reduction on 9/14 benchmarks
(0%~61%)
Computer Science
Study 2: Impact of domain reduction
RQ 2.1. #branches that have irrelevant parameters and %parameters that are irrelevant for each such branch?
RQ 2.2. %effort reduction for each such branch?
RQ 2.3. %effort reduction for each program?
Computer Science
RQ 2.1: Impact of domain reduction
90/434 aspectual branches with irrelevant parameters
Computer Science
RQ 2.1: Impact of domain reduction
Input domain reduction for branches with non-0 reduction
Input domain reduction (25%~100%)
Computer Science
RQ 2.2: Impact of domain reduction
Effort reduction per branch of using domain reduction
Effort increase on 25%, same on 6%, and reduction on 69% branches
94%
-88%
Easy/trivial branches
Computer Science
RQ 2.3: Impact of domain reduction
Effort reduction per program of using domain reduction
Effort reduction (17%~93%)
Computer Science
Findings: Impact of domain reduction
RQ 2.1. #branches that have irrelevant parameters (99/434) and %parameters that are irrelevant for each such branch (25%~100%)?
RQ 2.2. %effort reduction for each such branch (-88%~94%)(69% branches get reduction)?
RQ 2.3. %effort reduction for each program (17%~93%)?
Computer Science
Study 3: Impact of focusing on testing aspectual behavior
RQ 3.1. %effort reduction for test data generation if aspectual behavior instead of all behavior is focused on?
Computer Science
RQ 3.1: Impact of aspect focusing
Effort reduction of focusing on aspectual behavior over all behavior
Effort reduction on all 14 benchmarks
99.99%
3%
Computer Science
RQ 3.1: Impact of aspect focusing cont.
Coverage improvement of focusing on aspectual behavior over all behavior
Coverage improvement on 6/14 benchmarks
62%
Computer Science
Study 3: Impact of focusing on testing aspectual behavior
RQ 3.1. %effort reduction for test data generation if aspectual behavior instead of all behavior is focused on?
•Effort reduction on all 14 benchmarks (3% ~ 99.99%)•Coverage improvement on 6/14 benchmarks (0% ~ 62%)
Computer Science
Conclusion
• A new system of automated test data generation for AOP based on Search-Based Testing
• Empirical studies to demonstrate the benefits of the system in AOP structural testing– Effectiveness: better than random testing
– Efficiency: AOP domain reduction techniques
– Efficiency: focusing test effort on aspectual branches to improve efficiency
• Future work on more advanced techniques (e.g., symbolic execution), more testing objectives, larger AOP programs
Computer Science
Questions?
Computer Science
Level 4
Level 3
Level 2
Level 1
Fitness = Approximation_Level + Local_Distance
101 = 0 + (100 – 0) + 1
51 = 0 + (100 – 50) + 1
0 = 0 + (100 – 101) + 1
Evaluation of predicate in a branching condition
if (A < B) Local_Distance = (A – B) + 1
Identify relevant branching statements using control dependence, e.g.,(#expected/#actual – 1)
Target
Approximation levelApproximation levelApproximation levelApproximation level
Structural Evolutionary TestingStructural Evolutionary Testing
Local distanceLocal distanceLocal distanceLocal distance
Lower fitness value, the better 0 fitness value, reach the target
Computer Science
RQ 2.4: Impact of domain reduction
Co-lateral coverage improvement effect of domain reduction
9 branches have statistically significant change in co-lateral coverage