CS4723 Lecture 3Unit Testing

2

Unit testing

Testing of an basic module of the software A function, a class, a component

Typical problems revealed Local data structures

Algorithms

Boundary conditions

Error handling

3

Unit test framework

xUnit Created by Kent Beck in 1989

This is the same guy we mentioned in XP, design patterns

The first one was sUnit (for smalltalk)

Junit The most popular xUnit framework

There are about 70 xUnit frameworks for corresponding languages

Never in the annals of software engineering was so much owed by so many to so few lines of code

-------Martin Fowler

4

An example: writing test cases without a test framework

Class Definition Consider a class that handles and manages a Fibonacci

array

Initialized the class Fibonacci f = new Fibonacci ();

Extend to a certain length f. extend (length)

Provide the number at certain index f. get(index)

Get Length f.getLength()

Provide a range of numbers f. getRange(start, end)

5

An example: writing automatic test cases without a test framework

public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 0){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}

Not Enough!What happened if

an exception is thrown

6

An example: writing automatic test cases without a test framework

public static void main(String args[]){ Fibonacci f = new Fibonacci(); if(f.getLenght != 0){ System.err.print(“wrong results …”); } f.extend(50); if(f.getLenght != 0){ System.err.print(“wrong results …”); } int num = f.get(50); if(num != 12586269025){ System.err.print(“wrong results …”); } int[] range = f.getRange(40, 50) if(!ValueEquals(range, …)){ System.err.print(“wrong results …”); } …}

Potential test interference!

Consider if the get method does an extension when length is smaller than index:

while in getRange, such extension is forgotten to be added

7

Common Operations

Comparison between actual results and expected results

Report error messages

Catch Exceptions

Re-initialize the class to be tested

8

Unit Test Framework

System under test: the system/module/component we are testing

Test Fixture: SUT + DOC

Test Method: The actual code of the test

Test Case: A collection of tests with common purpose/setup

9

Writing a Test Casepublic class FibonacciTest extends TestCase { protected Fibonacci fTest; protected static int groudtruth[] = {0, 1,...,12586269025}; public void setUp(){ fTest = new Fibonacci(); } @Test public testInitialize(){ assertEquals(“Length of fTest after initialization”, 0, fTest.getLength()); } @Test public testExtend(){ fTest.extend(50); assertEquals(“…”, 50, fTest.getLength()); } @Test public testGet(){ fTest.extend(50); assertEquals(“…”, groudtruth[49], fTest.get(50)); } …}

10

Benefits of using test framework

Clear structure of testing Each test method for one feature

Common initialization to setup

Assertion methods You can always define new assertion methods

Reduced Interference Try catch to run all test methods

Always do re-initialization in setup

11

Tips of writing test cases

Bottom Up If you know that obj.m1 calls obj.m2, test obj.m2 first

Why? You may not understand the error in m1 if you don’t

know whether m2 is working or not

Externalize data Externalize expected results to files and load them at

the test case constructor

Why? Easier to maintain if you want to reuse them in several

places

12

Tips of writing test cases

Use development Database Do not use real database for testing

Why? Obvious

Do enough logging in the code during testing Why?

Help debugging

Testing for exceptions How?

public testException(){

try{

do(invalid); fail();

}catch(xxxException e){}

}

13

Writing Assertions

public void testCapacity() { // a test method

….

assertTrue(fFull.size() == 100+size); //assertion

}

If assertion fails:Assertion failed: myTest.java:150 (expected true but was

false)Not so good!

Try: assertEquals(100+size, fFull.size()); //expected value firstAssertion failed: myTest.java:150 (expected 102 but was

103)Better!

Try: assertEquals(“list length”, 100+size, fFull.size()); Assertion failed: myTest.java:150 (list length expected 102

but was 103)

14

Assertions

Extend TestCase and write your own assertions AssertStringContains, AssertArrayEquals, …

Use fail(String) to fail a test with certain message

Feel free to change the fixture Each test will re-setup

public void testRemoveAll() {

fFull.removeAllElements();

fEmpty.removeAllElements();

assertTrue(fFull.isEmpty());

assertTrue(fEmpty.isEmpty());

}

15

State machine of JUnit

Initialization Of Test Class

setUptry{ testMethod} catch {…}

tearDown

Tear down Consider the following test code

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}

Problems?

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { try { use f and b } finally { clean&close f, b }}void testBBB() { try { use f and b } finally { clean&close f, b }}

Better?

17

Tear down Consider the following test code

void setUp() { File f = open(“foo”); File b = open(“bar”);}void testAAA() { use f and b}void testBBB(){ use f and b}void tearDown{ clean&close f, b}

Problems?

void setUp() { File f = open(“foo”); File b = open(“bar”);}…void tearDown{ try{ clean&close f, b }catch{ … }}

void setUp() { File f = open(“foo”); File b = open(“bar”);}…void tearDown{ try{ clean&close f }catch{ … } the same for b}

Better?

18

Tear down

Do some cleanup job after the test is executed Close a file

Clear a global data structure

Revert changes to the database

Close network connections

…

19

Tear down

Be careful about tear down If tear down is not complete, a test failure may affect

the following test cases

Recover the changes done to global data that are not well handled by the setup Database, files, network, global variables

Clean resources

Caution of exceptions in tear down itself

20

The Practical Difficulty in Unit Testing

Decoupling It is rare that you are writing test cases for a class

with no dependencies (e.g., fibonacci array)

Dependencies are everywhere Complex structures System calls Databases File systems Network …

21

Problem of dependencies

Maintenance during setUp and tearDown

Expensive to setUp Network, loading from file, start a database…

Hard to find bugs Bugs may be related to the dependencies

They are still bugs, but should not be considered until system testing

22

Handling dependencies

Explicitly show dependencies Hidden dependencies are dangerous

Dependency injection

Remove unnecessary dependencies Reasonable design

Double dependencies When dependency is large or not ready yet

Use mock object

23

A story from google

A new developer go to a group and want to try on some methods

testCreditCardCharge() { CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008); c.charge(100);}

This code:Only works when you run as part of the suite.When run in isolation, throws NullPointerException.When you get your credit card bill, you are out $100 for every time the test runs.

24

A story from google

After a lot of digging, you learn that you need to initialize the CreditCardProcessor.testCreditCardCharge() { CreditCardProcessor.init(); CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008); c.charge(100);}

Still not working

25

A story from google

After a lot of digging, you learn that you need to initialize the OfflineQueue.testCreditCardCharge() { OfflineQueue.init(); CreditCardProcessor.init(); CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008); c.charge(100);}

Still not working

26

A story from google

After a lot of digging, you learn that you need to initialize the Database.testCreditCardCharge() { Database.init(); OfflineQueue.init(); CreditCardProcessor.init(); CreditCard c = new CreditCard( "1234 5678 9012 3456", 5, 2008); c.charge(100);}

But sometimes you have to find out the correct sequence!

27

Solution with dependency injection

testCreditCardCharge() { Database db = Database(); OfflineQueue q = OfflineQueue(db); CreditCardProcessor ccp = new CreditCardProcessor(q); CreditCard c = new CreditCard("1234 5678 9012 3456", 5, 2008); c.charge(ccp, 100);}

28

Lessons Learned

Always try to make dependencies explicitly shown

Never do testing with real credit card number,

Or at least try $1 first

29

Remove Dependencies Thin method interfaces

Example: Get the total rent of a car

getRent(car, customer) -> getRent(car.getDays(), car.getPrice(), customer.getDiscount())

Duplicate common dependencies Example:

The class for Graduate students and under-graduate students both depend on a calendar module

It says, “if(student.isGraduate){…}else{…}” Split it to two parts and put them into the class for

both types of students

30

Double Dependencies Removing Dependencies is not always doable

Lots of necessary dependencies

It is mostly something in design, so maybe unchangeable at the testing time

It is not always good to remove all unnecessary dependencies increase maintenance effort reduce the benefit of software reuse

make code more complex and hard to read

31

Double Dependencies

Solution: Double it!

Double: highly trained replacement, vaguely resembling the actor No need to be a good actor

May require different skills for different cases

32

Types of test doubles

Dummies

Test stubs

Fake objects

Mock objects

33

Dummies

Simplest test double No behavior, just serve as an argument

Null can be viewed as a simple double

Examplepublic class Order{ private Shop sp; public Order (Shop sp){ this.sp = sp; } public Shop getShop(){ return sp; } …}

public class ShopDummy extends Shop{ ...}

public class OrderTest{ @Test public void testInit(){ Order o = new order(new ShopDummy()); ... }}

When to use dummies

SUT does not invoke any method on the dependency

The dependency class is an interface or abstract class with no implementation

or

The dependency class is hard to initialize so that you can put the initialization in the dummy once E.g., Shop sp = new Shop(new A(…), new B(…), new

C(…), new D(…), new E(…));

Can put the whole thing in ShopDummy

35

Test Stubs

More powerful than dummies

Provide a fix value or fixed behavior for a certain method invocation

Example: Always return 0 for a integer method

Do nothing for a void method

36

Hard-coded Test Stubs

The value or behavior is hard coded in the Stub Class

public class OrderTest{ @Test public void test(){ Order o = new order(new ShopStub()); o.add(1122, 3); ... AssertEquals(expect, o.getTotal()); o.save(); }}

public class ShopStub extends Shop{ public void save(Order o){ } public double getShopDiscount(){ return 0.9; }}

37

Configurable Test Stubs You may set different values for different test

casespublic class ShopStub extends Shop{ private Exception saveExc; private discount; public setException(Exception e){ this.saveExc = e; } public setDicount(Float f){ this.discount = f; } public void save(Order o){ if(this.saveExc!=null){throw saveExc;} } public double getShopDiscount(){ return 0.9; }}

public class OrderTest{ @Test public void testAbnormalDiscount(){ ShopStub stub = new ShopStub(); stub.setDiscount(1.1); Order o = new order(stub); o.add(1122, 3); ... AssertEquals(expect, o.getTotal()); o.save(); }}

Fake Objects

More powerful than stubs

A simplified implementation of the DOC Example: a data table to fake a database

Example: use a greed algorithm to fake a complex optimized algorithm

Guidelines Slow -> Fast

Complex -> Simple

Fake Objects

Tips for fake objects As simple as possible (as long as not too time-

consuming)

Go to a higher level if some object is hard to fake Example:

URLStatus sts = HttpConnection.open("http://api.dropbox.com/files/myfile");if(sts.status == 200){ return sts.data;}else{ return “Error”;}

Need to double1. Difficult to reproduce2. Maybe slow3. Affected by lots of

factors

Fake Objects Example (1) Fake HttpConnection

We need to fake URLStatus also

public class FakeUrlStatus{ public FakeUrlStatus(int status, string data){...}; public int getStatus(){...}; public String getData(){...};}

public class FakeDropBoxApi{ private files = { }; public FakeUrlStatus read(fname){ if(files.contain(fname)){ return FakeUrlStatus(200, files[fname]); }else{ return FakeUrlStatus(-1, "Error"); } }}

FakeURLStatus sts = FakeDropBoxApi.read("myfile");if(sts.status == 200){ …

Fake Objects Example (2) Fake the HttpConnection + URLStatus

public class FakeDropBoxHigherApi{ private files = { }; public String read(fname){ if(files.contain(fname)){ return files[fname]; }else{ return "Error"; } }}

return FakeDropBoxHigherApi.read("myfile");

Fake Objects

Choose a reasonable abstraction level to do the faking Do not need to fake lots of dependencies (put them in

one box)

Write simpler code (less potential bugs)

43

Mock objects

Mock objects is not similar to any of the above For the ability to imitate DOC, it is similar to

configurable stubs

But it does some different things

Mock objects do behavior-based testing Usually we only check return values or status

AssertEquals (expected, actual); AssertEquals (expected, array.length);

Can we do something like this? Why? Assert ( testObject.f1 calls DOC.f)

44

Mock objects

Go back to the order example

44

public class OrderTest{ @Test public void test(){ Order o = new order(new ShopStub()); o.add(1122, 3); ... AssertEquals(expect, o.getTotal()); o.save(); }}

public class ShopStub extends Shop{ public void save(Order o){ } public double getShopDiscount(){ return 0.9; }}

Problems???

Mock objects

Do behavioral assertions for SUT Only cares about whether the unit to be tested is

correct

Do not care about the correctness of DOC

Test Code

SUT DOC

DON’TCARE!!

Mock objects

Example EasyMock

@Testpublic void testOrder() { Order o = new Order(sp); o.add(1234, 1); o.add(4321, 3);

//initialize Shop sp = EasyMock.CreateMock(Shop.class);

//record EasyMock.expect(sp.getDiscount()).andReturn(0.9); EasyMock.expect(sp.save(o));

//replay EasyMock.replay(sp); AssertEquals(expect, o.getTotal()); o.Save(); EasyMock.verify(sp)}

47

Mock objects

Procedure

I am a mock object, just generated

Record:What I am expected to be invoked, and what should I returnExpect Methods

Replay:See how SUT invoked meSUT

Verification Results

48

Verification

Verifies whether the expected methods are actually invoked Exception: missing, expected save(0xaaaa)

Verifies whether the expected methods are invoked in an expected way Exception: save(null) expected save(0xaaaa)

49

Richer ways to do expectation

Different frameworks support it in different ways

Easymock: isA: ignore the value of argument

EasyMock.expect(sp.save(isA(Order.class)))

Find: expect the argument to contain a certain substring EasyMock.expect(mock.call(find(“pattern”)))

Geq: expect a number larger than the given value EasyMock.expect(mock.call(Geq(1000)))

50

Orders of expected method calls

Different frameworks support it in different ways

Easymock Order is not checked by default createStrictMock to check order

Mockito Order is not checked by default Create a InOrder object at the verification phase and add

the expected invocations that should happen in order

jMock More complex order control

51

Fluent Mocking

Record phase: Read expectation as specifications

Instead of directly generating code, the mock object generates an internal presentation, e.g. Automaton

Replay phase: Check the real invocations with the internal

presentation

52

Fluent Mocking: jMock

53

Types of test doubles

Dummies Simplest, only for arguments

Test stubs Return fixed value

Can be configurable at runtime

Fake objects A simplified implementation

Mock objects Similar to configurable stubs

Behavior checking

Automatic framework support

54

Review of Unit Testing

Using unit test framework to do unit testing It does all the common things, reduce test

interference

Inside test framework setUp -> test -> assert -> tearDown

Writing informative assertions

Writing complete tear downs

Dependencies are evil! Dependency injection

Remove dependencies

Test doubles

55

Announcement of Assignment I

Task Writing unit test cases for a real-world Java class

One of two classes from the Apache project org.apache.commons.email.Email org.apache.commons.imaging.ImageParser

Write test method for only the listed 12 methods

Work individually

56

Announcement of Assignment I

Deliverable Send the test code, and required dependencies to my

email account

Dependencies include JUnit Mock Framework if used The tested code and required dependencies

Comment your test code properly

Due Feb 28th 2014

57

Announcement of Assignment I

Evaluation (20 points total) Test code compiles and has no runtime error (failing

test cases are not counted as runtime errors) – 8 points

Proper setUp, tearDown, and assertions – 4 points

Proper test doubles – 4 points

Good test coverage – 4 points

58

Demo

Writing a JUnit testcase for a class in your project

Run it as a JUnit test case