using dynamic compilers for software testing ben breech lori pollock john cavazos
TRANSCRIPT
![Page 1: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/1.jpg)
Using Dynamic Using Dynamic Compilers for Software Compilers for Software
TestingTesting
Ben BreechBen BreechLori PollockLori Pollock
John CavazosJohn Cavazos
![Page 2: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/2.jpg)
Background: Static CompilersBackground: Static Compilers
sourcecode
Lexical, syntactic, Semantic analysis
Inter. Rep.(basic blocks)
Static AnalysisOptimization
Code generationExecutable
Code
Static Compiler
![Page 3: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/3.jpg)
Background: Background: Software TestingSoftware Testing
Assesses Assesses qualityquality of software of software• Correctness, performance, securityCorrectness, performance, security
Most testing performed by executing codeMost testing performed by executing code
ExecutableCode
TestInput
ExecuteProgram
ProgramResults
ExpectedResults
Check Results(oracle)
![Page 4: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/4.jpg)
Motivating ExampleMotivating Exampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {
findmem ();findmem ();if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }
How do I test this callsite?How do I test this callsite?
Make the machine run out of memory?Make the machine run out of memory?Flip the conditional, recompile, flip back?Flip the conditional, recompile, flip back?
Pretend it doesn’t exist during testing?Pretend it doesn’t exist during testing?
![Page 5: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/5.jpg)
Generalizing the ProblemGeneralizing the Problem
Code to handle Code to handle uncommon situationsuncommon situations• Difficult to testDifficult to test• May need external environment event to triggerMay need external environment event to trigger
Examples:Examples:• Error handling codeError handling code• Testing program security mechanismsTesting program security mechanisms
![Page 6: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/6.jpg)
ObservationObservation
Hard to reach code executes when program Hard to reach code executes when program thinksthinks something uncommon has occurred something uncommon has occurred
if ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) { findmem ();findmem ();
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
Could test Could test findmemfindmem() by simulating error() by simulating error• E.g., could add instructions to program so program E.g., could add instructions to program so program
believes believes mallocmalloc failed failed
![Page 7: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/7.jpg)
RUGRAT ApproachRUGRAT Approach
Use Use Dynamic CompilersDynamic Compilers to generate to generate test cases for hard to reach code.test cases for hard to reach code.
AutomaticallyAutomatically add instructions to add instructions to program program during executionduring execution to simulate to simulate uncommon situation.uncommon situation.
![Page 8: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/8.jpg)
Dynamic CompilersDynamic Compilers
Dynamic compilers perform compilation Dynamic compilers perform compilation tasks tasks duringduring program execution program execution
code
Analysis&
optimization
Create basic block
translate
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
![Page 9: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/9.jpg)
RUGRAT ArchitectureRUGRAT Architecture
code
Analysis&
optimization
Create basic block
translate
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
Create basic block
Dynatest Generator
Testspec
Test OracleTest
Report
![Page 10: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/10.jpg)
Dynatest GeneratorDynatest Generator
Decides when/where/how to add testDecides when/where/how to add test• Decision based on test specificationDecision based on test specification
Test spec examplesTest spec examples• ““test all test all mallocsmallocs in program” in program”• ““test 2nd test 2nd mallocmalloc call in function call in function foofoo””
![Page 11: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/11.jpg)
ExampleExampleif ((sptr = malloc (size + 1)) == NULL) {if ((sptr = malloc (size + 1)) == NULL) {
findmem ();findmem ();
xlfail (“insufficient string space”); }xlfail (“insufficient string space”); }
if ((sptr = malloc (size + 1)) == NULL)if ((sptr = malloc (size + 1)) == NULL)
call malloc (code for malloc)movl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …
Dynatest Generator
call malloc (code for malloc)movl 0, <return val>movl ENOMEM, errnomovl <return val> sptrcmpl sptr, 0jnz L1call findmem….L1: …
![Page 12: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/12.jpg)
Good Times, Bad TimesGood Times, Bad Times
The Bad: Not a perfect simulationNot a perfect simulation
The Good: Adequate simulationAdequate simulation Can target system or appl callsCan target system or appl calls Saves quite a lot of tester effortSaves quite a lot of tester effort
![Page 13: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/13.jpg)
Security Mechanism Testing:Security Mechanism Testing:Encrypting Function PointersEncrypting Function Pointers
Protects progs against func pointer attacksProtects progs against func pointer attacks Difficult to test (need vulnerable program and Difficult to test (need vulnerable program and
attack)attack) RUGRAT can simulate attack by adding RUGRAT can simulate attack by adding
instructionsinstructions• Very different from error handling code caseVery different from error handling code case
RUGRAT can be used for variety of RUGRAT can be used for variety of testing tasks.testing tasks.
![Page 14: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/14.jpg)
Experiments SummaryExperiments Summary
Tested variety programs with RUGRATTested variety programs with RUGRAT 120+ error code handling callsites covered120+ error code handling callsites covered
• Both application and system callsBoth application and system calls Increased error code coverage ~ 50% over Increased error code coverage ~ 50% over
regular test casesregular test cases• Not all error code statements could be coveredNot all error code statements could be covered
• Different options, etcDifferent options, etc
Reasonable time overheadReasonable time overhead
![Page 15: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/15.jpg)
Some related workSome related work
Holodeck [1], FIG [2] Holodeck [1], FIG [2] • Require tester provide alternative “stub” Require tester provide alternative “stub”
functions to do testingfunctions to do testing• Miss application callsMiss application calls
Dynamic branch switching [3]Dynamic branch switching [3]• Not originally intended for testing error codeNot originally intended for testing error code• Need to know which branch to changeNeed to know which branch to change• Far less accurate simulationFar less accurate simulation
[1] Thompson et al., SAC 2002[1] Thompson et al., SAC 2002[2] Broadwell et al., SHAMAN 2002[2] Broadwell et al., SHAMAN 2002[3] Zhang et al., ICSE 2006[3] Zhang et al., ICSE 2006
![Page 16: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/16.jpg)
Conclusions and SummaryConclusions and Summary
Presented RUGRAT architecturePresented RUGRAT architecture• Can test hard to reach (and seldom tested) Can test hard to reach (and seldom tested)
code by using dynamic compilerscode by using dynamic compilers• Saves tester effortSaves tester effort
RUGRAT is a general toolRUGRAT is a general tool
![Page 17: Using Dynamic Compilers for Software Testing Ben Breech Lori Pollock John Cavazos](https://reader036.vdocuments.mx/reader036/viewer/2022062519/5697c01c1a28abf838ccfd7d/html5/thumbnails/17.jpg)
RUGRAT ArchitectureRUGRAT Architecture
code
Basicblock
Mod. Basicblock
Executeon CPU
Dynamic Compiler
Create basic block
Dynatest Generator
Testspec
Test OracleTest
Report