representations and operators for improving evolutionary software repair claire le goues westley...
TRANSCRIPT
![Page 1: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/1.jpg)
http://genprog.cs.virginia.edu 1
REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR
Claire Le Goues
Westley Weimer
Stephanie Forrest
![Page 2: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/2.jpg)
Claire Le Goues, GECCO 2012 2
PROBLEM: BUGGY SOFTWARE
http://genprog.cs.virginia.edu
“Everyday, almost 300 bugs appear […] far too many for only the Mozilla programmers to handle.”
– Mozilla Developer, 2005
Annual cost of software errors in the
US: $59.5 billion (0.6% of GDP).
Average time to fix a security-critical error:
28 days.
90%: Maintenance
10%: Everything Else
![Page 3: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/3.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 3
APPROACH: EVOLUTIONARY COMPUTATION
![Page 4: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/4.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 4
Genetic Programming
Search
Genetic Programming
Search
Input: source code, specification
Output: repaired version of the program
Input: source code, specification
Output: repaired version of the program
![Page 5: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/5.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 5
SEARCH SPACE
![Page 6: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/6.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 6
Learning: patches or repaired programs
Population: 40
Iterations: 10
Max variant size: unbounded
Largest benchmark program: 2.8 million lines of C code
Learning: patches or repaired programs
Population: 40
Iterations: 10
Max variant size: unbounded
Largest benchmark program: 2.8 million lines of C code.
OTHER GP PROBLEMSGECCO GP-TRACK
BEST PAPERS
Learning: expression trees or lists
Population: 64 – 2500
Iterations: 50 – 10000
Max variant size: 16 operations, 48 operations, 17 levels, 11 levels
PROGRAM REPAIR
![Page 7: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/7.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 7
SEARCH SPACE
EC-based repair starts with a large genome.
The starting individual is mostly correct.
![Page 8: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/8.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 8
Genetic Programming
Search
Input: source code, specification
Output: repaired version of the program
![Page 9: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/9.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 9
IN-DEPTH STUDY OF REPRESENTATION AND
OPERATORS FOR EVOLUTIONARY PROGRAM
REPAIR.
OUR GOAL
IN-DEPTH STUDY OF REPRESENTATION AND
OPERATORS FOR EVOLUTIONARY PROGRAM
REPAIR.
![Page 10: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/10.jpg)
Claire Le Goues, GECCO 2012
INPUT
OUTPUT
EVALUATE FITNESS
DISCARD
ACCEPT
10CROSSOVER, MUTATE, SELECT
![Page 11: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/11.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 11
IN-DEPTH STUDY OF REPRESENTATION AND
OPERATORS FOR EVOLUTIONARY PROGRAM
REPAIR.
OUR GOAL
![Page 12: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/12.jpg)
Claire Le Goues, GECCO 2012
INPUT
OUTPUT
EVALUATE FITNESS
DISCARD
ACCEPT
12CROSSOVER, MUTATE, SELECT
![Page 13: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/13.jpg)
Claire Le Goues, GECCO 2012CROSSOVER, MUTATE, SELECT
DISCARD
INPUT EVALUATE FITNESS
ACCEPT
OUTPUT
![Page 14: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/14.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 14
5 6
8 910
11
12
7
1
2 3 4
![Page 15: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/15.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 15
Input:
REPRESENTATION
Patch:
Delete(3)
Replace(3,5)
AST/WP:
1
2
4 5
5’
2
4
1
5
54
1
2 3 Delete(3) Insert(2,4) Replace(5,1)
Insert(5,4) Insert(3,3) Delete(4) …
![Page 16: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/16.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 16
Mutation operators
• Manipulate only existing genetic material.• Semantic checking improves probability that mutation is
viable.
Crossover:
• One-point: on the weighted path or edit list.• Patch-subset: uniform, on the edit list.
Mutation operators
• Manipulate only existing genetic material.• Semantic checking improves probability that mutation is
viable.
Crossover:
• One-point: on the weighted path or edit list.• Patch-subset: uniform, on the edit list.
GENETIC OPERATORSswap insertdelete
Aside: mutation operator selection matters!
![Page 17: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/17.jpg)
Claire Le Goues, GECCO 2012
Legend:
Likely faulty. probability
Maybe faulty. probability
Not faulty.
2
5 6
1
3 4
8
7
9
11
10
12
http://genprog.cs.virginia.edu
Input:
17
2
5 6
1
3 4
8
7
9
11
10
12
Legend:
High change probability.
Low change probability.
Not changed.
Default: 10 : 1 ratio
![Page 18: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/18.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 18
IN-DEPTH STUDY OF REPRESENTATION AND
OPERATORS FOR EVOLUTIONARY PROGRAM
REPAIR.
OUR GOAL
![Page 19: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/19.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 19
Benchmarks: 105 bugs in 8 real-world programs.1
• 5 million lines of C code, 10,000 test cases.• Bugs correspond to human-written repairs for regression test
failures.
Default parameters, for comparison:
• Patch representation.• Mutation operators selected with equal random probability. 1
mutation, 1 crossover/individual/iteration. • Population size: 40. 10 iterations or 12 wall-clock hours,
whichever comes first. Tournament size: 2.
EXPERIMENTAL SETUP
1Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer, “A Systematic Study of Automated Program Repair: Fixing 55 out of 105 bugs for $8 Each.” International Conference on Software Engineering (ICSE), 2012, pp. 3 – 13.
![Page 20: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/20.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 20
55/105 bugs repaired using default parameters.
Some bugs are more difficult to repair than others!
• Easy: 100% success rate on default parameters.• Medium: 50 – 100% success rate on default parameters• Hard: 1 – 50% success rate on default parameters• Unfixed: 0% success
Metrics:
• # fitness evaluations to a repair • GP success rate.
EXPERIMENTAL SETUP
![Page 21: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/21.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 21
Representation:
• Which representation choice gives better results? • Which representation features contribute most to
success?
Crossover: Which crossover operator is best?
Operators:
• Which operators contribute the most to success? • How should they be selected?
Search space: How should the representation weight program statements to best define the search space?
RESEARCH QUESTIONS
Representation:
• Which representation choice gives better results? • Which representation features contribute most to
success?
Crossover: Which crossover operator is best?
Operators:
• Which operators contribute the most to success? • How should they be selected?
Search space: How should the representation weight program statements to best define the search space?
Representation:
• Which representation choice gives better results? • Which representation features contribute most to
success?
Crossover: Which crossover operator is best?
Operators:
• Which operators contribute the most to success? • How should they be selected?
Search space: How should the representation weight program statements to best define the search space?
![Page 22: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/22.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 22
Procedure:
Compare AST/WP to PATCH on original benchmarks with default parameters.
For both representations, test effectiveness of: 1. Crossover.
2. Semantic check.
Results:
1. Patch outperforms AST/WP (14 – 30%).
2. Semantic check strongly influences success rate of both representations.
3. Crossover also improves results.
REPRESENTATION: RESULTS
Procedure:
Compare AST/WP to PATCH on original benchmarks with default parameters.
For both representations, test effectiveness of: 1. Crossover.
2. Semantic check.
Results:
1. Patch outperforms AST/WP (14 – 30%).
2. Semantic check strongly influences success rate of both representations.
3. Crossover also improves results.
![Page 23: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/23.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 23
Representation:
• Which representation choice gives better results? • Which representation features contribute most to
success?
Crossover: Which crossover operator is best?
Operators:
• Which operators contribute the most to success? • How should they be selected?
Search space: How should the representation weight program statements to best define the search space?
RESEARCH QUESTIONS
![Page 24: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/24.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 24
CROSSOVER: RESULTS
Crossover Operator Success Rate
Fitness evaluations to repair
None 54.4% 82.43
Default/“Uniform” 61.1% 163.05
One-Point/AST-WP
63.7% 114.12
One-Point/Patch 65.2% 118.20
Crossover Operator Success Rate
Fitness evaluations to repair
None 54.4% 82.43
Default/“Uniform” 61.1% 163.05
One-Point/AST-WP
63.7% 114.12
One-Point/Patch 65.2% 118.20
Crossover Operator Success Rate
Fitness evaluations to repair
None 54.4% 82.43
Default/“Uniform” 61.1% 163.05
One-Point/AST-WP
63.7% 114.12
One-Point/Patch 65.2% 118.20
![Page 25: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/25.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 25
Representation:
• Which representation choice gives better results? • Which representation features contribute most to
success?
Crossover: Which crossover operator is best?
Operators:
• Which operators contribute the most to success? • How should they be selected?
Search space: How should the representation weight program statements to best define the search space?
RESEARCH QUESTIONS
![Page 26: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/26.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 26
SEARCH SPACE: SETUP
Hypothesis: statements executed only by the failing test case(s) should be mutated more often than those also executed by the passing test cases.
Procedure: examine that ratio in actual repairs.
Result:
Expected: 10 : 1
vs.
Actual: 1 : 1.85
Hypothesis: statements executed only by the failing test case(s) should be mutated more often than those also executed by the passing test cases.
Procedure: examine that ratio in actual repairs.
Result:
![Page 27: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/27.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 27
Easy Medium Hard All0
10
20
30
40
50
60
70
80
90
100
110
34.3
93.6103.1
66.0
49.1
75.7
27.1
59.5
36.3
67.057.7 58.6
DefaultRealisticEqual
Search difficulty
# fi
tnes
s ev
alu
atio
ns
to r
epai
rSEARCH SPACE: REPAIR TIME
![Page 28: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/28.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 28
Easy Medium Hard 0% All0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%1.00
0.78
0.24
0.00
0.62
0.930.85
0.230.23
0.69
0.92 0.90
0.33
0.19
0.70
Default
Realistic
Equal
Search difficulty
GP
Su
cces
s R
ate
SEARCH SPACE: SUCCESS RATE
![Page 29: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/29.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 29
This EC problem is atypical; atypical problems warrant study.
We studied representation and operators for EC-based bug repair. These choices matter, especially for difficult bugs.
• Incorporating all recommendations, GenProg repairs 5 new bugs; repair time decreases by 17–43% on difficult bugs.
We don’t know why some of these things are true, but:
• We now have lots of interesting data to dig into!• We are currently (as we sit here) doing more and bigger
runs with the new parameters on the as-yet unfixed bugs.
CONCLUSIONS/DISCUSSION
![Page 30: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/30.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 30
PLEASE ASK QUESTIONS
![Page 31: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/31.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 31
REPRESENTATION BENCHMARKSProgram LOC Tests Bug Description
gcd 22 6 infinite loop Example
uniq-utx 1146 6 segfault Text processing
look-utx 1169 6 segfault Dictionary lookup
look-svr 1363 6 infinite loop Dictionary lookup
units-svr 1504 6 segfault Metric conversion
deroff-utx
2236 6 segfault Document processing
nullhttpd 5575 7buffer
exploitWebserver
indent 9906 6 infinite loop Source code processing
flex 18775 6 segfault Lexical analyzer generator
atris 21553 3buffer
exploitGraphical tetris game
Total 63249
![Page 32: REPRESENTATIONS AND OPERATORS FOR IMPROVING EVOLUTIONARY SOFTWARE REPAIR Claire Le Goues Westley Weimer Stephanie Forrest](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649f445503460f94c65109/html5/thumbnails/32.jpg)
Claire Le Goues, GECCO 2012 http://genprog.cs.virginia.edu 32
SEARCH SPACE BENCHMARKSProgram LOC Tests Bugs Description
fbc 97,000 773 3 Language (legacy)
gmp 145,000 146 2 Multiple precision math
gzip 491,000 12 5 Data compression
libtiff 77,000 78 24 Image manipulation
lighttpd 62,000 295 9 Web server
php 1,046,000 8,471 44 Language (web)
python 407,000 355 11 Language (general)
wireshark 2,814,000 63 7 Network packet analyzer
Total 5,139,000 10,193 105
55/105 bugs repaired using default parameters.