josé campos yan gen gordon fraser marcelo eler andrea …josé campos, yan gen, gordon fraser,...
TRANSCRIPT
![Page 1: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/1.jpg)
September 11th, 20179th Symposium on Search-Based Software Engineering (SSBSE)
Paderborn, Germany
An Empirical Evaluation of EvolutionaryAlgorithms for Test Suite Generation
José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri
University of São Paulo
![Page 2: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/2.jpg)
Class Under Test
![Page 3: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/3.jpg)
Test Suite optimised for structural coverage
![Page 4: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/4.jpg)
Research Questions
RQ1 - Which evolutionary algorithm works best for test suite optimisation?
RQ2 - How does evolutionary search compare to random search and random testing?
RQ3 - How does evolution of whole test suites compare to many-objective optimisation of test cases?
![Page 5: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/5.jpg)
Standard GA(1 out of 5 EAs)
![Page 6: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/6.jpg)
Standard GA(1 out of 5 EAs)
![Page 7: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/7.jpg)
Initial Population
@Testpublic void test0() { int var0 = 10; YearMonthDay var1 = new YearMonthDay(var0); TimeOfDay var2 = new TimeOfDay(); DateTime var3 = var1.toDateTime(var2); DateTime var4 = var3.minus(var0); DateTime var5 = var4.plusSeconds(var0);}
@Testpublic void test1() { DateTime var0 = new DateTime("11-09-2017"); DateTime var1 = new DateTime("25-12-2017"); int var2 = DateTime.sub(var0, var1);}
© Fraser and Arcuri, Whole Test Suite Generation, TSE 2013
Population
![Page 8: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/8.jpg)
Standard GA(1 out of 5 EAs)
![Page 9: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/9.jpg)
public Complex log() {1 if (isNaN) {2 return NaN; }
3 double r = log(abs());4 double i = atan2(imaginary, real);5 return createComplex(r, i); }
public Complex pow(double x) throws NullArgumentException {6 Complex c = this.log();7 return c.multiply(x).exp(); }
© Rojas, Campos, Vivantii, Fraser, Arcuri, Combining Multiple Coverage Criteria in Search-Based Unit Test Generation, SSBSE 2015
Fitness Evaluation(line coverage)
T1 T2 T3 T4
![Page 10: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/10.jpg)
public Complex log() {1 if (isNaN) {2 return NaN; }
3 double r = log(abs());4 double i = atan2(imaginary, real);5 return createComplex(r, i); }
public Complex pow(double x) throws NullArgumentException {6 Complex c = this.log();7 return c.multiply(x).exp(); }
© Rojas, Campos, Vivantii, Fraser, Arcuri, Combining Multiple Coverage Criteria in Search-Based Unit Test Generation, SSBSE 2015
T1⚫⚫
T2⚫
⚫⚫
T3⚫
⚫⚫
⚫⚫
T4⚫
⚫⚫
⚫ ⚫
2/7 3/7 6/7 4/7
Fitness Evaluation(line coverage)
![Page 11: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/11.jpg)
Standard GA(1 out of 5 EAs)
![Page 12: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/12.jpg)
Standard GA(1 out of 5 EAs)
![Page 13: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/13.jpg)
Crossover(exchanging test cases)
![Page 14: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/14.jpg)
Crossover(exchanging test cases)
![Page 15: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/15.jpg)
Standard GA(1 out of 5 EAs)
![Page 16: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/16.jpg)
Mutation(adding a new test case)
![Page 17: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/17.jpg)
Mutation(modifying an existing test case)
![Page 18: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/18.jpg)
Standard GA(1 out of 5 EAs)
![Page 19: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/19.jpg)
Monotonic GA(2 out of 5 EAs)
Population
Parents
Offspring
2/7 6/7
3/7 1/7
![Page 20: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/20.jpg)
Steady-State GA(3 out of 5 EAs)
Population
Parents
Offspring
![Page 21: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/21.jpg)
1 + ( , ) GA(4 out of 5 EAs)
Population
![Page 22: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/22.jpg)
1 + ( , ) GA(4 out of 5 EAs)
Population
mutants
![Page 23: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/23.jpg)
1 + ( , ) GA(4 out of 5 EAs)
Population
mutants offspring
2/7
3/7
![Page 24: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/24.jpg)
1 + ( , ) GA(4 out of 5 EAs)
Population
mutants offspring
2/7
3/7
6/7
1/7
4/7
![Page 25: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/25.jpg)
+ Evolutionary Algorithm(4 out of 5 EAs)
mutants
![Page 26: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/26.jpg)
+ Evolutionary Algorithm(4 out of 5 EAs)
mutants
2/7 3/7
![Page 27: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/27.jpg)
Many-Objective Sorting Algorithm (MOSA)(5 out of 5 EAs)
![Page 28: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/28.jpg)
Many-Objective Sorting Algorithm (MOSA)(5 out of 5 EAs)
![Page 29: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/29.jpg)
Many-Objective Sorting Algorithm (MOSA)(5 out of 5 EAs)
![Page 30: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/30.jpg)
Many-Objective Sorting Algorithm (MOSA)(5 out of 5 EAs)
![Page 31: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/31.jpg)
Random Search Test Generation(no selection, crossover, or mutation)
Population
![Page 32: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/32.jpg)
Random Search Test Generation(no selection, crossover, or mutation)
Population
![Page 33: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/33.jpg)
Random Search Test Generation(no selection, crossover, or mutation)
PopulationPopulation
2/7 3/7
![Page 34: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/34.jpg)
Random Test Generation
Population
![Page 35: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/35.jpg)
Random Test Generation
Population
![Page 36: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/36.jpg)
Random Test Generation
Population
2/7 +1
Population
3/7
![Page 37: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/37.jpg)
Enough of theory! Showme some results.
![Page 38: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/38.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
![Page 39: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/39.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
1 + (λ, λ) GA
µ + λ EAStandard GA, Steady State GA, MOSA, DynaMOSA, Random-search, and Random-testing
![Page 40: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/40.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
Tuning experiment /Larger study
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
1 + (λ, λ) GA
µ + λ EAStandard GA, Steady State GA, MOSA, DynaMOSA, Random-search, and Random-testing
![Page 41: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/41.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
34 / 312Tuning experiment /Larger study
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
1 + (λ, λ) GA
µ + λ EAStandard GA, Steady State GA, MOSA, DynaMOSA, Random-search, and Random-testing
![Page 42: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/42.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
34 / 312 60s & 600sTuning experiment /Larger study
Single &Multiple-criteria
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
1 + (λ, λ) GA
µ + λ EAStandard GA, Steady State GA, MOSA, DynaMOSA, Random-search, and Random-testing
![Page 43: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/43.jpg)
Experimental Setup
[17] Panichella, Kifetew, Tonella, Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets, TSE 2017
34 / 312 60s & 600sTuning experiment /Larger study
Single &Multiple-criteria
30 repetitions &10 repetitions
CUTs from DynaMOSA study [17]
346 Java classes from 117 open-source projects
1,109 statements and 259 branches on average
1 + (λ, λ) GA
µ + λ EAStandard GA, Steady State GA, MOSA, DynaMOSA, Random-search, and Random-testing
![Page 44: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/44.jpg)
Parameter Tuning
![Page 45: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/45.jpg)
Parameter Tuning*
* Random-based approaches do not require any tuning十[5] Doerr, Doerr, Ebel, From black-box complexity to designing new genetic algorithms, Theoretical Computer Science 2015土 [13] Jansen, De Jong, Wegener, On the choice of the offspring population size in evolutionary algorithms, Evolutionary Computation 2005
Population
Standard GA, Monotonic GA,Steady-State GA, MOSA, DynaMOSA 10, 25, 50, 100
1 + (λ, λ) GA 1, 8十, 25, 50
µ + λ EA 1, 7土, 25, 50
![Page 46: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/46.jpg)
Best Parameters
Population Coverages Avg.
10 73 100 13 21 43 6 98 62 12 100 52.8
25 80 79 73 62 24 81 46 81 84 53 66.3
50 54 78 35 26 20 7 90 59 25 4 39.8
What population size allows EA A to achieve the highest coverage of class C ?
![Page 47: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/47.jpg)
Best Parameters
Population Coverages Avg.
10 73 100 13 21 43 6 98 62 12 100 52.8
25 80 79 73 62 24 81 46 81 84 53 66.3
50 54 78 35 26 20 7 90 59 25 4 39.8
Â10,25 = 0.39 p-value = 0.44Â10,50 = 0.60 p-value = 0.29Â25,50 = 0.76 p-value = 0.04
What population size allows EA A to achieve the highest coverage of class C ?
![Page 48: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/48.jpg)
Best Parameters
Single-criteria Multiple-criteria
Algorithm 1 minute 10 minutes 1 minute 10 minutes
Standard GA 10 100 100 25
Monotonic GA 25 100 100 25
Steady-State GA 100 10 100 25
1 + (λ, λ) GA 50 50 50 8
µ + λ EA 1+7 50+50 1+7 1+1
MOSA 100 50 25 10
DynaMOSA 25 25 --- ---
![Page 49: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/49.jpg)
Best Parameters
Single-criteria Multiple-criteria
Algorithm 1 minute 10 minutes 1 minute 10 minutes
Standard GA 10 100 100 25
Monotonic GA 25 100 100 25
Steady-State GA 100 10 100 25
1 + (λ, λ) GA 50 50 50 8
µ + λ EA 1+7 50+50 1+7 1+1
MOSA 100 50 25 10
DynaMOSA 25 25 --- ---
![Page 50: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/50.jpg)
Best Parameters
Single-criteria Multiple-criteria
Algorithm 1 minute 10 minutes 1 minute 10 minutes
Standard GA 10 100 100 25
Monotonic GA 25 100 100 25
Steady-State GA 100 10 100 25
1 + (λ, λ) GA 50 50 50 8
µ + λ EA 1+7 50+50 1+7 1+1
MOSA 100 50 25 10
DynaMOSA 25 25 --- ---
![Page 51: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/51.jpg)
Best Parameters
Single-criteria Multiple-criteria
Algorithm 1 minute 10 minutes 1 minute 10 minutes
Standard GA 10 100 100 25
Monotonic GA 25 100 100 25
Steady-State GA 100 10 100 25
1 + (λ, λ) GA 50 50 50 8
µ + λ EA 1+7 50+50 1+7 1+1
MOSA 100 50 25 10
DynaMOSA 25 25 --- ---
![Page 52: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/52.jpg)
RQ1 - Best Evolutionary Algorithm
![Page 53: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/53.jpg)
RQ1 - Best Evolutionary Algorithm
![Page 54: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/54.jpg)
RQ1 - Best Evolutionary Algorithm
![Page 55: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/55.jpg)
RQ1 - Best Evolutionary Algorithm
![Page 56: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/56.jpg)
RQ1 - Best Evolutionary Algorithm
![Page 57: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/57.jpg)
RQ1 - Best Evolutionary Algorithm
In 3 out of 4 configurations, µ + λ EA is better than the other considered evolutionary algorithms.
![Page 58: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/58.jpg)
RQ2 - Evolutionary Search vs. Random
![Page 59: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/59.jpg)
RQ2 - Evolutionary Search vs. Random
![Page 60: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/60.jpg)
RQ2 - Evolutionary Search vs. Random
![Page 61: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/61.jpg)
RQ2 - Evolutionary Search vs. Random
![Page 62: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/62.jpg)
RQ2 - Evolutionary Search vs. Random
Evolutionary algorithms (in particular µ + λ EA) perform better than random search and testing.
![Page 63: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/63.jpg)
RQ3 - Whole test suites vs. MOSA
![Page 64: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/64.jpg)
RQ3 - Whole test suites vs. MOSA(single-criteria)
![Page 65: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/65.jpg)
RQ3 - Whole test suites vs. MOSA(multiple-criteria)
![Page 66: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/66.jpg)
RQ3 - Whole test suites vs. MOSA(multiple-criteria)
MOSA improves over EAs for individual criteria; for multiple-criteria it achieves higher branch
coverage even though overall coverage is lower.
![Page 67: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/67.jpg)
The choice of algorithm can have a substantial influence on the performance of test suite optimisation, hence tuning is important. While EvoSuite provides tuned default values, these values may not be optimal for different flavours of evolutionary algorithms.
![Page 68: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/68.jpg)
EvoSuite’s default algorithm, a Monotonic GA, is an appropriate choice for EvoSuite’s default configuration (1 minute search budget, multiple criteria). However, for other search budgets and optimisation goals, other algorithms such as aµ + λ EA may be a better choice.
![Page 69: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/69.jpg)
Our study shows that on complex classes evolutionary algorithms are superior to random testing and random search.
![Page 70: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/70.jpg)
The Many Objective Sorting Algorithm (MOSA) is superior to whole test suite optimisation for single criteria. (It would be desirable to add support to all coverage criteria in DynaMOSA)
![Page 71: José Campos Yan Gen Gordon Fraser Marcelo Eler Andrea …José Campos, Yan Gen, Gordon Fraser, Marcelo Eler, Andrea Arcuri University of São Paulo. Class Under Test. Test Suite optimised](https://reader035.vdocuments.mx/reader035/viewer/2022071001/5fbe66e509a3f244ac6ec8c6/html5/thumbnails/71.jpg)
http://www.evosuite.org/