april 29, 2005 gcuu csc-299 graduate seminar 1 of 25 application reliability robert j. conti
DESCRIPTION
3 of 25April 29, 2005GCUU CSc-299 Graduate Seminar Today’s Agenda… Introduction A Step Back – What’s Def-Use Testing? Application Reliability Concluding Remarks Q & ATRANSCRIPT
![Page 1: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/1.jpg)
April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar 11 of 25 of 25
Application ReliabilityApplication ReliabilityRobert J. ContiRobert J. Conti
![Page 2: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/2.jpg)
22 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Prelude…Prelude…
If you thought designing and coding that program If you thought designing and coding that program was hard, you ain’t seen nothing yetwas hard, you ain’t seen nothing yet..
- GJ Myers, 1979GJ Myers, 1979
Testing of Java Web Services for RobustnessTesting of Java Web Services for Robustness. . Chen Fu, Barbara G. Ryder, Ana Milanova, & David Wonnacott. Chen Fu, Barbara G. Ryder, Ana Milanova, & David Wonnacott. ISSTA 2004.ISSTA 2004.
![Page 3: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/3.jpg)
33 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Today’s Agenda…Today’s Agenda…
IntroductionIntroductionA Step Back – What’s Def-Use Testing?A Step Back – What’s Def-Use Testing?Application ReliabilityApplication ReliabilityConcluding RemarksConcluding RemarksQ & AQ & A
![Page 4: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/4.jpg)
44 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Introduction – Drilling Down…Introduction – Drilling Down…
IOEx
Java Web
Services
Web Services
Application Reliability
![Page 5: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/5.jpg)
55 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Introduction - Java Exceptions…Introduction - Java Exceptions…
exceptionexception
throwthrow
trytry
catchcatch
void method1( ) {
... try {
method2(“a:\\file.txt”); }catch (IOException e) {
e.printStackTrace(System.out); }...
}
void method2(String file) throws IOException {
... InputStream in = new FileInputStream(file);int c = in.read( );...
}
![Page 6: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/6.jpg)
66 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
A Step Back – What’s def-use testing?A Step Back – What’s def-use testing?
Black-box testingBlack-box testing
White-box testingWhite-box testing Control-flowControl-flow
e.g., statemente.g., statement Data-flowData-flow
e.g., def-usee.g., def-use
![Page 7: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/7.jpg)
77 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
A Step Back – What’s def-use testing?A Step Back – What’s def-use testing?
Def-useDef-use3 occurrences3 occurrences defdef c-usec-use p-usep-use
![Page 8: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/8.jpg)
88 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
A Step Back – What’s def-use testing?A Step Back – What’s def-use testing?
Program graphProgram graph defdef c-usec-use p-usep-use
![Page 9: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/9.jpg)
99 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
A Step Back – What’s def-use testing?A Step Back – What’s def-use testing?
New family of path New family of path selection criteriaselection criteria
Bridges the gapBridges the gap
Why do we care?Why do we care?
![Page 10: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/10.jpg)
1010 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability - Metrics…Application Reliability - Metrics…
How can you gauge reliability?How can you gauge reliability? By using e-c links and test set TBy using e-c links and test set T Determine set of faults (i.e., Determine set of faults (i.e., IOExceptionsIOExceptions)) Determine F Determine F { set of e-c links } { set of e-c links } Determine E Determine E { set of e-c links covered by { set of e-c links covered by
T }T }
Overall Exception Def-Catch Coverage = E / FOverall Exception Def-Catch Coverage = E / F
![Page 11: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/11.jpg)
1111 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability - Metrics…Application Reliability - Metrics…
The big pictureThe big picture
F
E
|E| / |F|
Reliability Faults
Web Service
![Page 12: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/12.jpg)
1212 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability - Mendosus…Application Reliability - Mendosus…
Mendosus is a fault injection engineMendosus is a fault injection engine trytry F – insert methods to request fault F – insert methods to request fault catchcatch F – insert method to record call F – insert method to record call
stack (for E) and to cancel faultstack (for E) and to cancel fault Only one fault will be tested at a timeOnly one fault will be tested at a time The facilitator behind this is the e-c link The facilitator behind this is the e-c link
analysisanalysis
![Page 13: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/13.jpg)
1313 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability - Mendosus…Application Reliability - Mendosus…
![Page 14: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/14.jpg)
1414 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Part I
Part II
Application Reliability – e-c Link Analysis…Application Reliability – e-c Link Analysis…
Part I – Exception flow analysisPart I – Exception flow analysis ASTAST Call graphCall graph Compiler dataflow Compiler dataflow
equationsequations Defines e-c linksDefines e-c links Imprecise due to Imprecise due to
infeasible link infeasible link generation!generation!
![Page 15: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/15.jpg)
1515 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – e-c Link Analysis…Application Reliability – e-c Link Analysis…
Infeasible linksInfeasible links readFile( )readFile( ) readNet( )readNet( )
BufferInputStream
.fill()
![Page 16: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/16.jpg)
1616 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – e-c Link Analysis…Application Reliability – e-c Link Analysis…
Infeasible links Infeasible links (cont’d)(cont’d)
Twice the e-c Twice the e-c linkslinks
How to fix?How to fix?
readFile() readNet()
FilterInputStream.read()
BufferedInputStream.read()
BufferInputStream.fill()
FileInputStream.read() SocketInputStream.read()
![Page 17: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/17.jpg)
1717 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Part I
Part II
Application Reliability – e-c Link Analysis…Application Reliability – e-c Link Analysis…
Part II – Data reach analysisPart II – Data reach analysis Post pass filterPost pass filter Call graphCall graph Points-to graphPoints-to graph Removes Removes
infeasible links!infeasible links!
![Page 18: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/18.jpg)
1818 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – Empirical Results…Application Reliability – Empirical Results…
FTPDFTPD
JNFSJNFS
HaboobHaboob
MuffinMuffin
Web Service Application Benchmarks
![Page 19: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/19.jpg)
1919 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – Empirical Results…Application Reliability – Empirical Results…
Java Web Service (Benchmark)
Configuration
E-C Link AnalysisReliability Faults
(IOEx)
The setupThe setup
E/F
![Page 20: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/20.jpg)
2020 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – Empirical Results…Application Reliability – Empirical Results…
Configuration Part IException-Flow
Part IIData Reach Inlining
CHA
RTA
PTA
InPTA
PTA-DR
InPTA-DR
![Page 21: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/21.jpg)
2121 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – Empirical Results…Application Reliability – Empirical Results…
e-c Links and Coverage
Program InPTA-DR
FTPD 85%
JNFS 84%
Haboob 83%
Muffin 83%
Overall Exception Def-Catch Coverage
![Page 22: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/22.jpg)
2222 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Application Reliability – Empirical Results…Application Reliability – Empirical Results…
![Page 23: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/23.jpg)
2323 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Concluding Remarks…Concluding Remarks…
SummarySummary Java web services and Java web services and IOExceptionIOExceptionss White-box data-flow def-use testingWhite-box data-flow def-use testing Exception flow def-useException flow def-use Overall Exception Def-Catch CoverageOverall Exception Def-Catch Coverage MendosusMendosus e-c link analysis (E-F and DR)e-c link analysis (E-F and DR) CoverageCoverage
![Page 24: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/24.jpg)
2424 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
Concluding Remarks…Concluding Remarks…
ConclusionsConclusions Telephone SystemTelephone System Outside the boxOutside the box Compiler techniquesCompiler techniques Other Java programsOther Java programs To be continued…To be continued…
![Page 25: April 29, 2005 GCUU CSc-299 Graduate Seminar 1 of 25 Application Reliability Robert J. Conti](https://reader033.vdocuments.mx/reader033/viewer/2022052706/5a4d1b237f8b9ab0599962b7/html5/thumbnails/25.jpg)
2525 of 25 of 25April 29, 2005April 29, 2005 GCUU CSc-299 Graduate SeminarGCUU CSc-299 Graduate Seminar
?Questions & Answers…Questions & Answers…