![Page 1: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/1.jpg)
CS527: (Advanced) Topics in Software Engineering
Overview of Software Quality Assurance
Tao Xie
©D. Marinov, T. Xie
![Page 2: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/2.jpg)
Overview
• Why look for bugs?
• What are bugs?
• Where they come from?
• How to detect them?
![Page 3: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/3.jpg)
Some Costly “Bugs”
• NASA Mars space missions– Priority inversion (2004)– Different metric systems (1999)
• BMW airbag problems (1999)– Recall of >15000 cars
• Ariane 5 crash (1996)– Uncaught exception of numerical overflow– http://www.youtube.com/watch?v=kYUrqdUyEpI
• Your own (in)famous example?
![Page 4: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/4.jpg)
Economic Impact
• “The Economic Impact of Inadequate Infrastructure for Software Testing”NIST Report, May 2002
• $59.5B annual cost of inadequate software testing infrastructure
• $22.2B annual potential cost reduction from feasible infrastructure improvements
![Page 5: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/5.jpg)
Economic Impact – cont.
• Is Knight's $440 million glitch the costliest computer bug ever?– http://money.cnn.com/2012/08/09/technology/knight-expensive-co
mputer-bug/index.html
![Page 6: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/6.jpg)
Terminology• Anomaly• Bug• Crash• Defect• Error• Failure, fault • Glitch• Hangup• Incorrectness• …
![Page 7: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/7.jpg)
Dynamic vs. Static
• Incorrect (observed) behavior– Failure
• Incorrect (unobserved) state– Error, latent error
• Incorrect lines of code– Fault, defect
![Page 8: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/8.jpg)
“Bugs” in IEEE 610.12-1990
• Fault– Incorrect lines of code
• Error– Faults cause incorrect (unobserved) state
• Failure– Errors cause incorrect (observed) behavior
• Not used consistently in literature!
![Page 9: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/9.jpg)
Correctness
• Common (partial) properties– Segfaults, uncaught exceptions– Resource leaks– Data races, deadlocks– Statistics based
• Specific properties– Requirements– Specification
![Page 10: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/10.jpg)
Traditional Waterfall ModelRequirements
Analysis
DesignChecking
ImplementationUnit Testing
IntegrationSystem Testing
MaintenanceVerification
![Page 11: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/11.jpg)
Phases (1)
• Requirements– Specify what the software should do– Analysis: eliminate/reduce ambiguities,
inconsistencies, and incompleteness
• Design– Specify how the software should work– Split software into modules, write specifications– Checking: check conformance to requirements
![Page 12: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/12.jpg)
Phases (2)
• Implementation– Specify how the modules work– Unit testing: test each module in isolation
• Integration– Specify how the modules interact– System testing: test module interactions
• Maintenance– Evolve software as requirements change– Verification: test changes, regression testing
![Page 13: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/13.jpg)
Testing Effort
• Reported to be >50% of development cost [e.g., Beizer 1990]
• Microsoft: 75% time spent testing– 50% testers who spend all time testing– 50% developers who spend half time testing
![Page 14: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/14.jpg)
When to Test
• The later a bug is found, the higher the cost– Orders of magnitude increase in later phases– Also the smaller chance of a proper fix
• Old saying: test often, test early
• New methodology: test-driven development(write tests before code)
![Page 15: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/15.jpg)
Software is Complex
• Malleable
• Intangible
• Abstract
• Solves complex problems
• Interacts with other software and hardware
• Not continuous
![Page 16: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/16.jpg)
Software Still Buggy
• Folklore: 1-10 (residual) bugs per 1000 nbnc lines of code (after testing)
• Consensus: total correctness impossibleto achieve for (complex) software– Risk-driven finding/elimination of bugs– Focus on specific correctness properties
![Page 17: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/17.jpg)
Approaches for Detecting Bugs
• Software testing
• Model checking
• (Static) program analysis
![Page 18: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/18.jpg)
Software Testing
• Dynamic approach
• Run code for some inputs, check outputs
• Checks correctness for some executions
• Main questions– Test-input generation– Test-suite adequacy– Test oracles
![Page 19: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/19.jpg)
Other Testing Questions
• Maintenance
• Selection
• Minimization
• Prioritization
• Augmentation
• Evaluation
• Fault Characterization
• …
![Page 20: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/20.jpg)
Model Checking
• Typically hybrid dynamic/static approach
• Checks correctness for all executions
• Some techniques– Explicit-state model checking– Symbolic model checking– Abstraction-based model checking
![Page 21: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/21.jpg)
Static Analysis
• Static approach
• Checks correctness for all executions
• Some techniques– Abstract interpretation– Dataflow analysis– Verification-condition generation
![Page 22: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/22.jpg)
Comparison
• Level of automation– Push-button vs. manual
• Type of bugs found– Hard vs. easy to reproduce– High vs. low probability– Common vs. specific properties
• Type of bugs (not) found
![Page 23: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/23.jpg)
Soundness and Completeness
• Do we detect all bugs?– Impossible for dynamic analysis
• Are reported bugs real bugs?– Easy for dynamic analysis
• Most practical techniques and tools are both unsound and incomplete!– False positives– False negatives
![Page 24: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/24.jpg)
Analysis for Performance
• Static compiler analysis, profiling• Must be sound
– Correctness of transformation: equivalence
• Improves execution time• Programmer time is more important
• Programmer productivity– Not only finding bugs
![Page 25: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/25.jpg)
Combining Dynamic and Static
• Dynamic and static analyses equal in limit– Dynamic: try exhaustively all possible inputs– Static: model precisely every possible state
• Synergistic opportunities– Static analysis can optimize dynamic analysis– Dynamic analysis can focus static analysis– More discussions than results
![Page 26: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/26.jpg)
Current Status
• Testing remains the most widely used approach for finding bugs
• A lot of recent progress (within last decade) on model checking and static analysis– Model checking: from hardware to software– Static analysis: from sound to practical
• Vibrant research in the area
• Gap between research and practice
![Page 27: CS527: (Advanced) Topics in Software Engineering Overview of Software Quality Assurance Tao Xie ©D. Marinov, T. Xie](https://reader031.vdocuments.mx/reader031/viewer/2022032312/56649de75503460f94ae1311/html5/thumbnails/27.jpg)
Topics Related to Finding Bugs
• How to eliminate bugs?– Debugging
• How to prevent bugs?– Programming language design– Software development processes
• How to show absence of bugs?– Theorem proving– Model checking, program analysis