![Page 1: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/1.jpg)
Automatic test case generation for programs that are coded against interfaces and
annotations or use native code
Mainul IslamSupervisor: Dr. Christoph Csallner
December 12th, 2012
The University of Texas at Arlington
PhD Research Proposal
![Page 2: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/2.jpg)
2
Outline
• Problem Description• Motivation• Limitation of Current Approaches (with example)• Thesis Statement• Background• Solution Approach• Experimental Results• Plan of Action
![Page 3: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/3.jpg)
3
Problem Description
• Current state of the art tools are not very good at generating test cases when the code under test:
– requires additional pieces of code that are not yet part of the program.
– uses multiple interfaces, annotations or reflection.– imposes complex (type) constraints.– uses native code.
![Page 4: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/4.jpg)
4
Motivation
• Automatic test case generation is important• At the initial stage of any development the
implementation of some code may not be available• Existing techniques (such as Pex and Moles*) are
not very good at generating test cases when the code under test uses interfaces/annotations, multiple inheritance and native code.
• Significant number of Java programs use native code
* http://research.microsoft.com/en-us/projects/pex/
![Page 5: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/5.jpg)
5
Limitation of Current Techniques: Motivating Example 1
public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }
public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } if ( i.getClass().isAnnotationPresent(A.class) ) { // .. } }}
To reach this block of code ‘i’ must be an
instance of I, as well as an instance of J
To reach this block of code ‘i’ must be an
instance of I, as well as annotated with A
![Page 6: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/6.jpg)
6
Motivating Example 2 (Dependency on Native Code)
public class C { public native boolean isDivisible(int x, int y); public static void NativeTest(int a, int b) { boolean divisible = new C().isDivisible(a, b); if (divisible) { // … } }}
bool isDivisible(int x, int y) { if ( y*(x/y) == x ) return true; return false;}
Java code C++ code
To reach this block of code ‘a’ must be divisible by ‘b’
![Page 7: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/7.jpg)
7
Motivation: Patterns in Real World Program
![Page 8: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/8.jpg)
8
Pattern 1
R m(…, T t, …) { // … if ( … (t instanceof X) ) // …}
• ‘T’ an ‘X’ are non-compatible• ‘m’ is user-defined• At least one of {T, X} is an
interface• None of {T, X} is final• At least one of {T, X} is a user-type
![Page 9: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/9.jpg)
9
Thesis Statement
• We can generate automatic test inputs and systematically increase code coverage compared to existing techniques specially when the code under test:– requires additional pieces of code that are not yet part of
the program.– uses multiple interfaces, annotations or reflection.– imposes complex (type) constraints.– uses native code.
![Page 10: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/10.jpg)
10
Background: Symbolic Execution
• Systematically explore all feasible execution paths
• Initialize the input with symbolic values and execute the program over symbolic values
• At conditional statements check if either of the branches can be taken
• For each path get an accumulated path condition
C` = C S⋀ C` = C ⋀ ⌝S
S
true false
C
If (S) then … else …
![Page 11: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/11.jpg)
Dynamic Symbolic Execution by Exampletaken with permission from Nikolai Tillmann (Microsoft Research)
11
public static void TestMe(int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”);}
a == null
a[0] == 123
a.length > 0
F T
F T
F T
Constraints to Solve Input (a) Observed Constraints
null a == null
a != null {} a != null && !(a.length > 0)
a != null && a.length > 0 {0}
a != null && a.length > 0 && a[0] != 123
a != null && a.length > 0&& a[0] == 123
{123} a != null && a.length > 0 && a[0] == 123
Solve Execute
Choose Next Path
Done: No Path Left
![Page 12: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/12.jpg)
12
• Java defines a binary sub-type relation• If type B implements/extends type A then,– A is a direct super-type of B– B is a direct sub-type of A
Reflexive: A is also a subtype of itselfTransitive: if B is a subtype of A and C is a
subtype of B then C is also a subtype of A
Sub-/Supertype relation in Java
![Page 13: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/13.jpg)
13
• A class has one direct class super type and arbitrarily many interface super types.
• Exceptions:type object – has no direct super typetype null – has no direct sub type
Sub-/Supertype relation in Java
![Page 14: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/14.jpg)
14
• Introduce Dynamic Symbolic Mock Classes
Technique implemented on: Dsc1 – Dynamic Symbolic Execution Engine for Java Z32 – SMT solver (from Microsoft Research)
Our Solution Approach
1 - http://ranger.uta.edu/~csallner/dsc/index.html2 - http://z3.codeplex.com/
![Page 15: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/15.jpg)
15
Solution WorkflowDefault Input
values (0, null, …)
New Test Cases (and corresponding
mock classes )
Invoke DSE on given input values and collect path constraints
Map the constraint solver model to new test cases (and mock classes)
Invert one of the collected path constraints
(Map each reference type to a constraint literal and encode their properties to build
a constraint system)
More Paths
?
Add mock classes and map each of them to a constraint literal
Encode properties (e.g., subtype relation) of mock
classes in the constraint system
Stop
Constraint System
Satisfiable?
No
No
Yes
Yes
Constraint System
Satisfiable?
Yes
No
![Page 16: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/16.jpg)
16
Subtype Constraints
public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }
public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } // .. } }
A desired solution with new Type: M, to reach the code block
Object
I J C
null
Initial Types in the system
A
Annotation
M
Constraints:type(i) subtypeof Itype(i) != null typetype(i) subtypeof J
![Page 17: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/17.jpg)
17
Subtype Constraints
public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }
public class C { public static void foo(I i) { int x = i.m1(); // .. if ( i.getClass(). isAnnotationPresent(A.class) ) { // .. } }
A desired solution with new Type: M1, to reach the code block
Object
I J C
null
Initial Types in the system
A
Annotation
M1
Constraints:type(i) subtypeof I
type(i) subtypeof Atype(i) != null type
![Page 18: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/18.jpg)
18
Subtype Relation Matrix
public @interface A { /* … */ }public interface I { public int m1(); /* … */ }public interface J { public int m2(); /* … */ }
public class C { public static void foo(I i) { int x = i.m1(); if ( i instanceof J ) { J j = (J) i; int y = j.m2(); } // .. } }
null Object An I J C A M
0 null x x x x x x x x
1 Object x
2 An x x
3 I x x
4 J x x
5 C x x
6 A x x x
7 M x mAn mI mJ mC mA x
Solution: mI = true mJ = true mC = false mA = mAn = false
null Object An I J C A
0 null x x x x x x x
1 Object x
2 An x x
3 I x x
4 J x x
5 C x x
6 A x x x
null Object An
0 null x x x
1 Object x
2 An x x
![Page 19: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/19.jpg)
19
Experimental Results (1)
• Experiments are done on simplified version of real world code
• C# codes are translated manually
![Page 20: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/20.jpg)
20
Experimental Results (2)
• Experiments are done on original code• Randoop has several side effects
![Page 21: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/21.jpg)
21
Workflow: to handle Native CodeDefault Input
values (0, null, …)
New Test Cases
Invoke DSE on given input values and collect path constraints
Map the constraint solver model to new test cases
Invert one of the collected path constraints and build constraint system
More Paths
?
Native Code
Invoked?
Collect the constraints from the native code for current input
values
Convert the native code constraints to Java constraints
Stop
Constraint System
Satisfiable?
No
No
No
YesYes
Yes
![Page 22: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/22.jpg)
22
Technique to handle Native Code
In each iteration of the dynamic symbolic execution of a Java program:• Check if a native code call is invoked. If yes, start executing
the native code on current input values• Use any existing tool (such as Klee* for C++ code) to collect
the path constraints of the native code • Convert the constraints collected from the native code and
merge them with the constraints previously collected from the Java program
• Solve the whole constraint system using a Constraint solver and generate test case.
• http://klee.llvm.org/
![Page 23: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/23.jpg)
23
Initial Experiments
In JDK 1.6:• Total # types: 23799• Total # of types (have at least one native
method): 381• Total # of native methods: 2019
• http://klee.llvm.org/
![Page 24: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/24.jpg)
24
Plan of Actions
Time Action Plan (to do)Spring 2013 Implement and evaluate the
technique to handle native code.
Submit a short paper
Summer 2013 Internship (currently interviewing)
Join back to UTA in Fall 2013
Fall 2013 Detail experiments of the technique to handle native
code against the existing techniques.
Submit the experiment results to a journal
paper
Spring 2014 Write up and submit dissertation.
Ph.D. defense
![Page 25: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/25.jpg)
25
List of publications
1. Mainul Islam and Christoph Csallner. Generating Test Cases for Programs that are Coded Against Interfaces and Annotations (submitted)
2. Mainul Islam and Christoph Csallner. Dsc+Mock: A test case + mock class generator in support of coding against interfaces (In Proc. 8th International Workshop on Dynamic Analysis (WODA), co-located with International Symposium on Software Testing and Analysis (ISSTA), 2010)
![Page 26: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/26.jpg)
26
Thank You!
![Page 27: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/27.jpg)
27
Backup Slides
![Page 28: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/28.jpg)
28
Pattern 2
class P {T t, …}
class Q { M m(…, P p, …) { // … if ( … (p.t instanceof X) ) // …}
![Page 29: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/29.jpg)
29
Pattern 3
M m(…) { // … if ( … (..).foo() instanceof X ) // …}
![Page 30: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/30.jpg)
30
Related work
Mock classes inferred from Programmer-Written specification:
• EasyMock, jMock, Mockito• NMock• Google Mock• SynthiaMock
![Page 31: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/31.jpg)
31
Related work: example with EasyMock
public interface K extends I, J { /* … */ }public class Test { public void testFooIsInstanceOfJ() { K mock = createMock(K.class); expect(mock.m1()).andReturn(0); expect(mock.m2()).andReturn(0); replay(mock); C.foo(mock); verify(mock); }}
![Page 32: Automatic test case generation for programs that are coded against interfaces and annotations or use native code Mainul Islam Supervisor: Dr. Christoph](https://reader030.vdocuments.mx/reader030/viewer/2022032516/56649c7b5503460f9492e94c/html5/thumbnails/32.jpg)
32
0.5
u1
0.2
u2
0.6
u3
0.4
u4
0.8
u5
0.3 0.3
0.4
0.4
0.2
0.2
0.9
0.9
0.5
0.5
Visual similarity
Geo similarity