a test case + mock class generator for coding against interfaces

A Test Case + Mock Class Generator for Coding Against Interfaces

Mainul Islam, Christoph CsallnerSoftware Engineering Research Center (SERC)

Computer Science and Engineering DepartmentUniversity of Texas at Arlington, USA

February 07, 2011.

Published at: 8th International Workshop on Dynamic Analysis (WODA), 2010

Mock Class Generator for Coding Against Interfaces

2

• Test case generation• Dynamic Symbolic Execution• Mock Class• Motivating examples to generate Mock Class• Algorithm to generate Mock Class• Results• Demonstration

Talk Outline

Mock Class Generator for Coding Against Interfaces

3

• What is the goal behind testing ?

Test Case Generation

public void Test(int x) { if (x > 0) {

if (x == 1234567890) { throw new Exception(“Bug”); }

}}

• What can be a good test case to test this method ?

Probability to reach the bug (with a random test input) is: ½ * 1/2^31

Mock Class Generator for Coding Against Interfaces

4

• Explore all feasible execution paths • Initialize the input with symbolic

values and execute the program over symbolic values

• At conditional statements check if either of the branches can be taken (using a constraint solver)

• For each path we get an accumulated path condition

Symbolic Execution

If (s) then … else … C

falsetrue

C` = C ⋀ s C` = C ⋀⌝s

s

• Dynamic symbolic execution: Combine Concrete and Symbolic execution

This example is taken from the slides of Nikolai Tillman (with permission)

5

void TestMe (int a[]) { if (a == null) return; if (a.length > 0) if (a[0] == 123) throw new Exception(“Error”);}

Dynamic Symbolic Execution by Example

a == null

a.length > 0

a[0] == 123

T

T

T

F

F

F

Constraints to Solve Input Observed Constraints

null

{}

{0}

{123}

a != null && !(a.length > 0)

a == null

a != null && a.length > 0 &&a[0] != 123

a != null && a.length > 0 &&a[0] == 123

a != null

a != null && a.length > 0

a != null && a.length > 0 &&a[0] == 123

Solve Execute

Choose Next Path

a == null

Done: No path left

Mock Class Generator for Coding Against Interfaces

6

public interface A {// …

}

public class C { public void foo (A a, int x) {// …// …}

}

Introduction to Mock Class

Mock Class Generator for Coding Against Interfaces

7

When we might need to generate mock-classes ?

• At the initial stage of any development

Mock Class Generator for Coding Against Interfaces

8

• Test case generation is important• Interfaces are very important• Existing techniques are not very good at this

Pex - can generate an instance of a class that implements an interface that is

used in the code *

* http://research.microsoft.com/en-us/projects/pex/

Why is this important ?

Mock Class Generator for Coding Against Interfaces

9

Goal:

• Support interfaces in test case generation for object-oriented programs

Mock Class Generator for Coding Against Interfaces

10

Motivating Example 1public class C {

public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x); }}

}To reach this block of code, ‘a’ must be an instance of A, as well as an instance of B

public interface A {int m1(int x);

}

public interface B {int m2(int x);

}

Mock Class Generator for Coding Against Interfaces

11

Motivating Example 2public class K

implements A, B { // …

}

public class C { public void bar (A a, int x) { if (a == null) return; if ( !(a instanceof B) ) {

B b = (B) a; int z = b.m2(x); }}

}

K implements all of the interfaces referred by the bar

method, but fails reach the code block

public interface A {int m1 (int x);

}

public interface B {int m2 (int x);

}

Mock Class Generator for Coding Against Interfaces

12

Framework

• DSC – Dynamic Symbolic Execution Engine (for Java)

• Z3 – Constraint Solverhttp://research.microsoft.com/en-us/um/redmod/projects/z3/

Mock Class Generator for Coding Against Interfaces

13

Our approach to generate Mock Class

• Determining the type of the Mock Class• Generate meaningful method body if needed

Mock Class Generator for Coding Against Interfaces

14

Sub-/Supertype relation in Java

• Java defines a binary subtype relation• If type B implements/extends type A then,

- A is a direct super type of B - B is a direct sub type of A

Reflexive: A is also a subtype of itselfTransitive: if B is a subtype of A and C is a

subtype of B then C is also a subtype of A

Mock Class Generator for Coding Against Interfaces

15

Sub-/Supertype relation in Java (Cont.)

• A class has one direct class super type and arbitrarily many interface super types.

• Exceptions:type object – has no direct super typetype null – has no direct sub type

Mock Class Generator for Coding Against Interfaces

16

Subtype constraintsObject

A B C

null

Object

A B C

M

nullInitial types in the system

A desired solution: with new type M

public interface A {int m1 (int x);

}

public interface B {int m2(int x);

}

public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x); }}

}

type(a) subtypeof A type(a) != null type type(a) subtypeof B

Constraints:

Mock Class Generator for Coding Against Interfaces

17

Subtype Relation Matrixpublic interface A {

int m1 (int x);}

public interface B {int m2(int x);

}

public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x); }}

}

Null Object A B C M

0 Null x x x x x x

1 Object x

2 A x x

3 B x x

4 C x x

5 M x MA MB MC x

Null Object

0 Null x x

1 Object x

Null Object A B C

0 Null x x x x x

1 Object x

2 A x x

3 B x x

4 C x x

Null Object A B C M

0 Null x x x x x x

1 Object x

2 A x x

3 B x x

4 C x x

5 M x x x x

Solution: MA = true MB = true MC = false

18

Program execution

Found Solution?

No

Yes

Algorithm: Generating Mock Class

Dynamic Symbolic Data Structure Repair

Build Subtype Relation matrix (Generate the constraints)

NO_MC < MAX_MC

Generate Test Case

Introduce a new Mock Class, N0_MC++

Encode the Mock Class in the type system

Yes

Yes

Solution not found

No

Initialization: NO_MC = 0 MAX_MC = 2

Mock Class Generator for Coding Against Interfaces

19

Generating Method Body: for Example 1

public class M implements A, B { int m1 (int x){ return 0;}

int m2(int x){ return 0;}

}

Outline of the source code for Mock Class M

public interface A {int m1 (int x);

}

public interface B {int m2(int x);

}

public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x);

if (z >= 10) { //… }

}}

}

return 10;

Mock Class Generator for Coding Against Interfaces

20

Results

• We have run our solution on JMS (Java messaging system)

• These are all hand-written test casesInterfaces used from

JMS

Number of goals

Number of goals reached

w/o mock-classes

Number of goals reached

with mock-classes

Message 6 0 6ByteMessage 5 0 5MapMessage 4 0 4

StreamMessage 4 0 4TextMessage 5 0 5

ObjectMessage 4 0 4

Mock Class Generator for Coding Against Interfaces

21

Comparison with Pex: Motivating Example 1

public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x); }}

}

public interface A {int m1(int x);

}

public interface B {int m2(int x);

}

Goal 1

Goal 2

PEX reaches 1/2 goal(s)

DSC reaches 2/2 goal(s)

Mock Class Generator for Coding Against Interfaces

22

Future Work

• Extend the solution to support more types, e.g. ‘Array’

• Determine a upper bound to the number of mock classes needed to solve a system

• Evaluation

Mock Class Generator for Coding Against Interfaces

23

Future Work (Cont.)

public class M implements A, B { int m1 (int x){ return 0;}

int m2(int x){ return 0;}

}

Outline of the source code for Mock Class M

public class C { public void foo (A a, int x) { int y = a.m1(x); if (a instanceof B) {

B b = (B) a; int z = b.m2(x);

if (z >= 10) { //… }

}}

}

return 10;

• Extend method body generation for complex types

Mock Class Generator for Coding Against Interfaces

24

Demo

Mock Class Generator for Coding Against Interfaces

25

Thank you!