comprehensive evaluation of association measures for software fault localization
DESCRIPTION
Comprehensive Evaluation of Association Measures for Software Fault Localization. LUCIA , David LO, Lingxiao JIANG, Aditya BUDI Singapore Management University. Introduction. A Buggy Program. Automated Fault Localization. Test Cases. Failure. Where is the fault ?. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/1.jpg)
Comprehensive Evaluation of Association Measures for
Software Fault Localization
LUCIA, David LO, Lingxiao JIANG, Aditya BUDISingapore Management University
![Page 2: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/2.jpg)
Introduction
2
Where is the fault ?
A Buggy Program
Automated Fault
Localization
Candidate of suspicious
program elements
Test Cases Failure
![Page 3: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/3.jpg)
Fault Localization Techniques
Many fault localization techniques have been proposed.
One family of techniques: Spectrum-based fault localization (Reps et.al, 1997)
• Use program spectra (the representation of program behavior during execution)
3
![Page 4: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/4.jpg)
An example of spectrum
4
Test Cases
![Page 5: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/5.jpg)
Spectrum-based Fault Localization
• IdeaProgram element that frequently occurs in failed test case is likely to contain bug
• Example of existing measures :• Tarantula, Ochiai, etc.
5
![Page 6: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/6.jpg)
Spectrum-based Fault Localization
6
Test Cases
![Page 7: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/7.jpg)
Spectrum-based Fault Localization
– Tarantula (Jones and Harrold, 2005)
– Ochiai (Abreu et.al, 2007)
7
![Page 8: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/8.jpg)
Our Contributions #1
Tarantula & Ochiai model fault localization as the association between
• The execution of program elements with occurrence of fault
We model fault localization as the association between
• The execution or non-execution of program elements with occurrence of fault
8
![Page 9: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/9.jpg)
Modeling Fault Localization with Association MeasuresSuspiciousness score of a program element (e) is
defined using an association measure (M) as follows:
• A Non-control element M(EXECUTION(e), FAILURE)
• A Control elementMaximum of
M(EXECUTION(e), FAILURE) and M(NON-EXECUTION(children of e), FAILURE)
9
![Page 10: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/10.jpg)
Our Contributions #2
• Evaluate the accuracies of 20 association measures for fault localization.
• Evaluate their relative performance as compared to Tarantula and Ochiai.
10
![Page 11: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/11.jpg)
Existing Association Measures (Tan et.al, 2002, Geng and Hamilton, 2006, Cheng et.al.,2009)
11
Association M.1 Coefficient2 Odd Ratio3 Yule’s Q4 Yule ‘s Y5 Kappa6 J-Measure7 Gini Index8 Support9 Confidence10 Laplace
Association M.11 Conviction12 Interest13 Cosine14 Piatetsky-Shapiro15 Certainty Factor16 Added Value17 Collective Strength18 Jaccard19 Klosgen20 Information Gain
![Page 12: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/12.jpg)
Modeling Fault Localization with Association Measures
12
Block 1 Execute !ExecuteFailed 1 0Passed 3 0
Block 2 Execute !ExecuteFailed 1 0Passed 2 1
![Page 13: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/13.jpg)
Modeling Fault Localization with Association Measures
13
Block 1 Execute !ExecuteFailed 1 0Passed 3 0
Block 2 Execute !ExecuteFailed 1 0Passed 2 1
e.g. Cosine
A=Execute, B=FailedA=Execute, B=Failed
A=Not Execute B=Failed
![Page 14: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/14.jpg)
Modeling Fault Localization with Association Measures
14
![Page 15: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/15.jpg)
Evaluation using Siemens Dataset
• dad
15
Dataset LOC #Faulty Versions
No. of Test
CasesPrint_tokens 472 5 4030Print_tokens2 399 10 4115Replace 512 31 5542Schedule 292 9 2650Schedule2 301 10 2710Tcas 141 36 1608Tot_Info 440 19 1052
Total : 120 buggy versions
![Page 16: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/16.jpg)
Some measures are not as good as Ochiai and Tarantula.
16
![Page 17: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/17.jpg)
Some measures are comparable Ochiai and Tarantula.
17
![Page 18: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/18.jpg)
Improvement by Association Measure
18
![Page 19: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/19.jpg)
Percentage of Inspected Elements
19
![Page 20: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/20.jpg)
The Statistical Significance between Measures
20
Coefficient, Kappa, Confidence, Interest, Cosine, Added Value, Collective Strength , Jaccard, Klosgen,
Information Gain, Tarantula, Ochiai
Odd Ratio, Yule’s Q, Yule ‘s Y, Support, Laplace, Conviction
Certainty Factor
Piatetsky-Shapiro
Gini Index
J-Measure
![Page 21: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/21.jpg)
Summary of Findings
– Fifty percent of the association measures have good accuracies for fault localization (28-34% inspected block)
– Association measures that are statistically comparable with Ochiai and Tarantula are: Coefficient, Kappa, Confidence, Interest, Cosine , Added Value, Collective Strength, Jaccard, Klosgen, Information Gain
– Information Gain can localize more bugs as compare to Ochiai when 20-50% blocks are
inspected.
21
![Page 22: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/22.jpg)
Threats to Validity
• The effect of different granularity of instrumentation level (http://www.mysmu.edu/phdis2009/lucia.2009/Dataset.htm)
• Dataset for experimentation is not a large program
22
Future Work• Investigate large real program• Investigate the effectiveness of the measures
for different types of bug
![Page 23: Comprehensive Evaluation of Association Measures for Software Fault Localization](https://reader036.vdocuments.mx/reader036/viewer/2022062302/568165f5550346895dd91b7d/html5/thumbnails/23.jpg)
THANK YOU
23