Feed Back Directed Random Test Generation

Carlos Pacheco1, Shuvendu K. Lahiri2, Michael D. Ernst1, and Thomas Ball2

1MIT CSAIL, 2Microsoft Research

Presented By :Aliasgar Kagalwala

Outline:

1. Concept2. Technique3. Evaluation4. Related work5. Conclusion

Concept

• This paper incorporates technique that improves random test generation by incorporating feedbacks obtained from executing test inputs.

• The feedback obtained from executing sequence that guide the search towards sequences that yield new and legal object states.

• In puts creating redundant states are never extended they prune the search space.

• This input is checked against set of contracts and filters

Concept…

• The work addresses random generation of unit tests for object-oriented programs. Implemented the technique in (RANDOOP) .

• RANDOOP is fully automatic requires no input from the user except name of binary for .NET or class directory for the Java.

• RANDOOP has found serious errors in widely deployed commercial software.

Test Case:Test case for java.utilpublic static void test1() {LinkedList l1 = new LinkedList();Object o1 = new Object();l1.addFirst(o1);TreeSet t1 = new TreeSet(l1);Set s1 = Collections.unmodifiableSet(t1);// This assertion failsAssert.assertTrue(s1.equals(s1));

• Test case shows violation of equals contract.

• The set s1 returned by unmodifiable Set(Set) returns false for s1.equals(s1) this violates reflexivity of equals as specified in Suns API doc.

• The other error is in TreeSet(Collection) contsructor which failed to throw ClassCastException as required by specification

An obj oriented unit test consist of sequence of method calls that set up a state (such as creating and mutating obj)

Extending Sequence:m is a method with formal parameters of type T1,..TkSeqs is the list of sequenceVals is a list of values .either its primitive or return value s.i of the ith method call.

Feedback directed algorithm for sequence.

It builds seq incrementally It has four inputs.Selecting public method of classes for seq creation.randomSeqsAndVals() builds list of seq and values newSeq is result of applying extension operatorexecute method executes each method call in seq and checks contract .

List of default contracts checked by RANDOOP.

RANDOOP outputs two input sets nonErrorSeqs and ErrorSeqs as Junit /Nunit test along with assertion representing the contract checked.

Filtering• Determines which values of a sequence are extensible and should be used

as the input to next method call.• As a result of applying filter to a sequence s the filter may set some s.i

extensible flags to false, so that this value will not be used as input to a new method call.

• There are basically three filters that RANDOOP uses by default:• EQUALITY,NULL,EXCEPTION.

• Equality:• This filter uses equals() method to determine if the resulting obj has been

created.• The filter maintains a set allobjs of all extensible objects that have been

created by algorithm across all sequence of execution.• This heuristic prunes any object with the same abstract value as a

previously created value even if their concrete representation differ.• This might cause RANDOOP to miss an error if method calls on them

behave differently.

• NULL:• Null dereference exception occur in absence of null value in i/p it signifies

some internal problem with the method.• Null arguments are hard to detect statically because arguments in the

sequence themselves are output of other sequence.• Instead null filter checks the values computed by execution of specific

sequence.

• Exception:

Exception frequently correspond to pre condition of violation for a method.Extension of the sequence would lead to exception before the execution

completes.

Repetition.

Repeated calls to add may be necessary to reach the code that increases the capacity of container object.

Or Repeated calls may be required to create two equivalent objects that can cause

a method like equals to go down certain branches.

• Thus repetition is build in the generator. As follows• When generating a new sequence, with probabilityN, instead of appending

a single call of a chosen method m to create a new sequence, the generator appends M calls, where M is chosen uniformly at random between

0 and some upper limit max. (max andN are user-settable; the default values are max = 100 and N = 0:1.)

Evaluation.

Evaluation…Container class have been used to evaluate input generation technique.Four container classes: a binary tree (BinTree,154 LOC), a

binomialheap(BHeap, 355 LOC), a fibonacci heap (FibHeap, 286 LOC), and a red-black tree (TreeMap,

580 LOC). They compared the coverage achieved by six techniques.(1) model checking,(2) model checking with state matching,(3) model checking with abstract state matching, (4) symbolic execution (5) symbolic execution with abstract state matching, (6) undirected random generation.

Evaluation…

• For each < technique, container > pair we report the maximum coverage achieved , and the time when maximum coverage was reached as shown by experiment.

Checking API Contract:

Checking API Contract.• In this experiment, they used feedback-directed random generation,

undirected random generation, and systematic generation to create test suites for 14 widely-used libraries comprising a total of 780KLOC.

• To reduce the amount of test cases they had to inspect, they implemented a test runner called REDUCE.

• REDUCE only shows subset of failing test.• REDUCE partitions the failing test into equivalence classes, two test fall

into the same class if their execution leads to contract violation.

• They ran RANDOOP on a library, specifying all the public classes as targets for testing. Using RANDOOP's default parameters The output of this test suite was.??

Test cases generated. The size of the test suite (numberof unit tests) output by RANDOOP. Violation-inducing test cases. The number of violationinducingtest cases output by RANDOOP.REDUCE reported test cases. The number of violation inducingtest cases reported by REDUCEErrors. The number of distinct errors uncovered by theerror-revealing test cases. We count two errors as distinctif fixing them would involve modifying different sourcecode. Errors per KLOC. The number of distinct errors dividedby the KLOC count for the library.

Feedback-directed random generation

• Errors Discovered were

RANDOOPcreated a total of 4200 distinct violation-inducing test cases.Of those, REDUCE reported approximately 10% .Out of the 424tests that REDUCE reported, 254 were error-revealing. Theother 170 were illegal uses of the libraries

Undirected Random Testing• RANDOOP was tested using the same parameters, but disabling the user

of filters or contracts to guide generation.

• The result obtained was. Undirected generation did not find any errors in java.util or javax.xml, and

was unable to create the sequence that uncovered the infinite loop in System.Xml.

Regression and compliance testing

• feedback-directed random testing to find inconsistencies between different implementations of the same API.

• RANDOOP guesses observer methods using a simple strategy: a method is an observer if all of the following hold:

• (i) it has no parameters,• (ii) it is public and non-static• (iii) it returns values of primitive type (or String), and• (iv) its name is size,• count, length, toString, or begins with get or is.

Related Work:

• Automatic test generation is active research area we focus on input generation technique that create method sequence.

Related Work…

• Random Testing:• JCrasher : Creates test inputs using a parameter graph whose values can

serve a as an input parameters. RANDOOP uses component set of previously created sequence.

• Feedback directed test generation was introduced by the Eclat tool. Eclat's performance is sensitive to the quality of the sample execution given as an input to the tool. Since RANDOOP does not require a sample

execution, it is not sensitive to this parameter.

• An experimental comparison of Eclat and RANDOOP is an interesting avenue for the future work.

Related Work…

Systematic Testing:

• Bounded exhaustive generation has been implemented in Rostra and JPF and RANDOOP with some differences.

• An alternative to bounded exhaustive approach is symbolic execution implemented in Symtra etc.

• Check n Crash creates abstract constraints over input that cause exceptional behavior and uses constraint solver to derive test inputs.

• Combining Random and Systematic Testing.

• DART a symbolic execution approach that integrates random input generation.

• RANDOOP is closer to random-systematic spectrum though it is random input generator it uses systemization to be more effective.

Conclusion

• Feedback directed random testing scales to a large extend and finds errors quickly to heavily tested application.

• Combining random testing and systematic testing gives advantage of both.

• Notion of exploration using a component set or state matching when there are many object can be translated into exhaustive test domain.

References:

• C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proc. 29th ACM/IEEE International Conference on Software Engineering (ICSE), pages 75{84. IEEE,May 2007.

• W. Visser, C. S. Pasareanu, and R. Pel´anek. Test input generation for Java containers using state matching. In ISSTA, pages 37.48, July 2006.

• C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classication of test inputs. In ECOOP, pages 504.527, July 2005.

• T. Xie, D. Marinov, and D. Notkin. Rostra: A framework fordetecting redundant object-oriented unit tests. In ASE, pages 196.205, Nov. 2004.

• T. Xie, D. Marinov, W. Schulte, and D. Notkin. Symstra:A framework for generating object-oriented unit tests using symbolic execution. In TACAS, pages 365.381, Apr. 2005.

• Question????

• THANK YOU..