Unit, Regression, and Unit, Regression, and Behavioral TestingBehavioral Testing

Based On: Based On: Unit Testing with JUnit and CUnit Unit Testing with JUnit and CUnit

bybyBeth KirbyBeth Kirbybeth@apl.washington.edubeth@apl.washington.edu206-685-0310206-685-0310Dec 13, 2002Dec 13, 2002

Jules WhiteJules Whitejules@dre.vanderbilt.edujules@dre.vanderbilt.edu

One minute summaryOne minute summary

‘‘Code that isn’t tested doesn’t work’Code that isn’t tested doesn’t work’ Why test?Why test?

Code that isn’t regression tested suffers Code that isn’t regression tested suffers from code rot (breaks eventually)from code rot (breaks eventually) Why regression test?Why regression test?

A unit testing framework is unit & A unit testing framework is unit & regression testing on steroidsregression testing on steroids

What is unit testing?What is unit testing?

Testing is often divided into categories such as:Testing is often divided into categories such as: Unit testingUnit testing

Testing an isolatable ‘unit’ of code, usually a classTesting an isolatable ‘unit’ of code, usually a classe.g. AStacke.g. AStack

Integration testingIntegration testingTesting a module of code (e.g. a package)Testing a module of code (e.g. a package)Testing AStack + Analyzer, LStack + AnalyzerTesting AStack + Analyzer, LStack + Analyzer

Application testingApplication testingTesting the code as the user would see it (black box)Testing the code as the user would see it (black box)Testing DelimitersTesting Delimiters

Writing a unit test - exampleWriting a unit test - example

public void testAnalyzer() {      public void testAnalyzer() {                  Analyzer analyzer = new Analyzer (….);Analyzer analyzer = new Analyzer (….);analyzer.process (“{“);analyzer.process (“{“);analyzer.process (“}”); analyzer.process (“}”);

//checks to see if the delimiters matched//checks to see if the delimiters matchedassertTrue (analyzer.match ()); assertTrue (analyzer.match ());

}}

What is a testing framework?What is a testing framework?

A test framework provides reusable test A test framework provides reusable test functionality which:functionality which: Is easier to use (e.g. don’t have to write the Is easier to use (e.g. don’t have to write the

same code for each class)same code for each class) Is standardized and reusableIs standardized and reusable Provides a base for regression testsProvides a base for regression tests

Why formalize unit testing?Why formalize unit testing?

Unit testing has often been done, but in Unit testing has often been done, but in an ad hoc manneran ad hoc manner E.g. adding a main method to a class, E.g. adding a main method to a class,

which runs tests on the classwhich runs tests on the classAxiom:Axiom: Code that isn’t tested doesn’t workCode that isn’t tested doesn’t work ‘‘If code has no automated test case written If code has no automated test case written

for it to prove that it works, it must be for it to prove that it works, it must be assumed not to work.’ (Hightower and assumed not to work.’ (Hightower and Lesiecki)Lesiecki)

Why use a testing framework?Why use a testing framework?

Each class must be tested when it is Each class must be tested when it is developeddevelopedEach class needs a regression testEach class needs a regression testRegression tests need to have standard Regression tests need to have standard interfacesinterfacesThus, we can build the regression test Thus, we can build the regression test when building the class and have a better, when building the class and have a better, more stable product for less workmore stable product for less work

Regression testingRegression testingNew code and changes to old code can New code and changes to old code can affect the rest of the code baseaffect the rest of the code base ‘‘Affect’ sometimes means ‘break’Affect’ sometimes means ‘break’

I need to run tests on the old code, to I need to run tests on the old code, to verify it works – these are regression verify it works – these are regression teststestsRegression testing is required for a Regression testing is required for a stable, maintainable code basestable, maintainable code base

Regression testing and Regression testing and refactoringrefactoring

‘‘Refactoring is a technique to restructure Refactoring is a technique to restructure code in a disciplined way.’ code in a disciplined way.’ (Martin Fowler)(Martin Fowler)

Refactoring is an excellent way to break Refactoring is an excellent way to break code.code.Regression testing allows developers to Regression testing allows developers to refactor safely – if the refactored code refactor safely – if the refactored code passes the test suite, it workspasses the test suite, it works

Running automated testsRunning automated tests

The real power of regression tests happens The real power of regression tests happens when they are automatedwhen they are automated This requires they report pass/fail results in a This requires they report pass/fail results in a

standardized waystandardized way

Can set up jobs to Can set up jobs to Clean & check out latest build treeClean & check out latest build tree Run testsRun tests Put results on a web page & send mail (if tests fail)Put results on a web page & send mail (if tests fail)

JUnit & ant have code to do all of thisJUnit & ant have code to do all of this

Some backgroundSome backgroundeXtreme programming (XP) is a eXtreme programming (XP) is a ‘programming methodology that stresses ‘programming methodology that stresses (among other things) testing and refactoring(among other things) testing and refactoringThe Apache community provides open-The Apache community provides open-source software, including the Jakarta project source software, including the Jakarta project (server side java tools, e.g. tomcat (servlet (server side java tools, e.g. tomcat (servlet engine))engine))Ant is a build tool (like make) Ant is a build tool (like make) Part of the Jakarta projectPart of the Jakarta project Is becoming the de facto standard for java projectsIs becoming the de facto standard for java projects Is written in java, uses xml to specify build targetsIs written in java, uses xml to specify build targets

Stateful TestingStateful Testing

What is Stateful Testing?What is Stateful Testing?Example:Example:

public void testAnalyzer() {public void testAnalyzer() {Analyzer analyzer = new Analyzer(…..);Analyzer analyzer = new Analyzer(…..);analyzer.process (“{“);analyzer.process (“{“);assertEqual (“{”, analyzer.getStack().topOfStack());assertEqual (“{”, analyzer.getStack().topOfStack());analyzer.process (“}”);analyzer.process (“}”);

assertEqual( 0, analyzer.getStack().size() );assertEqual( 0, analyzer.getStack().size() );}}

What is Wrong with Stateful What is Wrong with Stateful Testing?Testing?

What happens if Analyzer is refactored to What happens if Analyzer is refactored to use a data structure other than a stack?use a data structure other than a stack?

Example:Example:public void testAnalyzer() {public void testAnalyzer() {

Analyzer analyzer = new Analyzer(…..);Analyzer analyzer = new Analyzer(…..);analyzer.process (“{“);analyzer.process (“{“);assertEqual (“{”, analyzer.getStack().topOfStack());assertEqual (“{”, analyzer.getStack().topOfStack());analyzer.process (“}”);analyzer.process (“}”);

assertEqual( 0, analyzer.getStack().size() );assertEqual( 0, analyzer.getStack().size() );}}

What is Wrong with Stateful What is Wrong with Stateful Testing?Testing?

Stateful testing does not allow the interface and Stateful testing does not allow the interface and implementation to vary.implementation to vary.Stateful testing tightly couples the testing code Stateful testing tightly couples the testing code to the implementation so that refactoring breaks to the implementation so that refactoring breaks the testing codethe testing codeStateful testing also prevents testing code to be Stateful testing also prevents testing code to be reused as new classes are added that reused as new classes are added that implement the same interfaceimplement the same interface e.g. a test targeted specifically for AStack cannot be e.g. a test targeted specifically for AStack cannot be

used by LStackused by LStack}}

Behavioral TestingBehavioral TestingTests should check the expected behavior (input Tests should check the expected behavior (input / output)/ output)Tests should be coded to the specification not Tests should be coded to the specification not the implementationthe implementation

Example:Example:

public void testAnalyzer() {public void testAnalyzer() {Analyzer analyzer = new Analyzer(……);Analyzer analyzer = new Analyzer(……);analyzer.process (“{“);analyzer.process (“{“);analyzer.process (“}”);analyzer.process (“}”);Boolean expected = new Boolean(true);Boolean expected = new Boolean(true);

expected.equals( analyzer.match () );expected.equals( analyzer.match () );}}

Mock ObjectsMock Objects

How do you isolate an object for unit How do you isolate an object for unit testing when it relies on other objects? testing when it relies on other objects?

Example:Example:public void testAnalyzer() {public void testAnalyzer() {

Analyzer analyzer = new Analyzer(……);Analyzer analyzer = new Analyzer(……);analyzer.process (“{“);analyzer.process (“{“);analyzer.process(“}”);analyzer.process(“}”);

assertEqual( true, analyzer.match ()); // A stack is used hereassertEqual( true, analyzer.match ()); // A stack is used here}}

Mock ObjectsMock Objects

Mock Objects are used to stand in for the Mock Objects are used to stand in for the real objects that the class relies onreal objects that the class relies on e.g Stack is replaced by a MockStack e.g Stack is replaced by a MockStack Mock Objects check to ensure that the Mock Objects check to ensure that the dependent object calls the correct dependent object calls the correct methods in the correct ordermethods in the correct orderThey can guarantee the input and output They can guarantee the input and output to the dependent object and isolate the to the dependent object and isolate the source of errorssource of errors

Mock ObjectsMock Objects

public void testAnalyzer() {public void testAnalyzer() {Analyzer analyzer = new Analyzer(new Analyzer analyzer = new Analyzer(new MockObjectFactory());MockObjectFactory());……....

}}

MockObjectFactory {MockObjectFactory {……..Stack make_stack () { return new MockStackObject ();}Stack make_stack () { return new MockStackObject ();}}}

Mock ObjectsMock ObjectsMockStackObject {MockStackObject {……..boolean popCalled = false;boolean popCalled = false;boolean pushCalled = false;boolean pushCalled = false;boolean pushCalledFirst = false;boolean pushCalledFirst = false;

public void push (String str) { public void push (String str) { pushCalled = true;pushCalled = true; pushCalledFirst = !popCalled;pushCalledFirst = !popCalled;}}public String pop() {public String pop() { popCalled = true;popCalled = true; return “{“;return “{“;}}

Mock ObjectsMock Objects

Example:Example:public void testAnalyzer() {public void testAnalyzer() { MockObjectFactory fact = new MockObjectFactory()MockObjectFactory fact = new MockObjectFactory()

Analyzer analyzer = new Analyzer(fact);Analyzer analyzer = new Analyzer(fact);analyzer.process (“{“);analyzer.process (“{“);analyzer.process (“}”);analyzer.process (“}”);assertTrue( fact.mockStack.pushCalledFirst());assertTrue( fact.mockStack.pushCalledFirst());assertTrue( fact.mockStack.popCalled ());assertTrue( fact.mockStack.popCalled ());

assertTrue( true, analyzer.match ()); // A MockStack is used hereassertTrue( true, analyzer.match ()); // A MockStack is used here}}

What is Junit?What is Junit?

JUnit is a regression testing framework written JUnit is a regression testing framework written by Erich Gamma and Kent Beck by Erich Gamma and Kent Beck It is found at It is found at www.junit.orgwww.junit.orgIt consists of classes that the developer can It consists of classes that the developer can extend to write a test – notably extend to write a test – notably junit.framework.TestCase - and related junit.framework.TestCase - and related structure to run and report the testsstructure to run and report the tests

Tool integrationTool integration

Java IDEsJava IDEs

Eclipse – Eclipse – http://www.eclipse.orghttp://www.eclipse.orgExtensive support for JUnitExtensive support for JUnit

Netbeans - Netbeans - http://www.netbeans.org/index.htmlhttp://www.netbeans.org/index.html

Idea - Idea - http://www.intellij.com/idea/http://www.intellij.com/idea/

Writing a testWriting a test

Directions can be found in the Junit Directions can be found in the Junit cookbook - cookbook - http://junit.sourceforge.net/doc/cookbook/cookbook.hthttp://junit.sourceforge.net/doc/cookbook/cookbook.htmmSoundbite summarySoundbite summary

1.1. Create an instance of Create an instance of TestCaseTestCase::2.2. Write methods which run your tests, calling them test<Foo>Write methods which run your tests, calling them test<Foo>3.3. Call a test runner with your testCall a test runner with your test4.4. When you want to check a value, call an assert method (e.g. When you want to check a value, call an assert method (e.g.

assertTrue()) and pass a condition that is true if the test assertTrue()) and pass a condition that is true if the test succeedssucceeds

Writing a test - exampleWriting a test - example

public void testAnalyzer() {      public void testAnalyzer() {                  Analyzer analyzer = new Analyzer(……);Analyzer analyzer = new Analyzer(……);analyzer.process (“{“);analyzer.process (“{“);analyzer.process (“}”);analyzer.process (“}”);Boolean expected = new Boolean(true);Boolean expected = new Boolean(true);

expected.equals( analyzer.match () );expected.equals( analyzer.match () );

}}

Specifying testsSpecifying tests

JUnitJUnit The framework will look in every class The framework will look in every class

given to it which implements TestCasegiven to it which implements TestCase It uses reflection to find all methods that It uses reflection to find all methods that

start with ‘test’ (e.g. testMoneyMoneyBag)start with ‘test’ (e.g. testMoneyMoneyBag) It runs each such method as a test caseIt runs each such method as a test case

Test suitesTest suites

In practice, you will want to run a group of In practice, you will want to run a group of related tests (e.g. all the tests for a class)related tests (e.g. all the tests for a class)To do so, group your test methods in a To do so, group your test methods in a class which extends TestCaseclass which extends TestCaseOverride the constructor and (if desired) Override the constructor and (if desired) the setUp and tearDown methodthe setUp and tearDown method

Test Suite exampleTest Suite examplePackage test.mypackage;Package test.mypackage;import junit.framework.TestCase;import junit.framework.TestCase;Import mypackage;Import mypackage;

Public class MyClassTest extends TestCase {Public class MyClassTest extends TestCase { public MyClassTest(String name) {public MyClassTest(String name) { super(name);super(name); public void setUp throws Exception {public void setUp throws Exception { // set up for each test// set up for each test }} public void tearDown throws Exception {public void tearDown throws Exception { // releases resources // releases resources }} public void testMyMethod throws Exception {public void testMyMethod throws Exception { // Run tests// Run tests }}}}

Test Suite methodsTest Suite methods

setUp – does initialization common to all setUp – does initialization common to all teststests It is run each time a test is run (can’t store It is run each time a test is run (can’t store

state in variables that are set in setUp)state in variables that are set in setUp)

tearDown – releases resources, e.g. tearDown – releases resources, e.g. database connections. database connections. Can also have other helper methods, just Can also have other helper methods, just don’t name them test<anything>don’t name them test<anything>

Running testsRunning testsNeed to call a runner run method, e.g Need to call a runner run method, e.g junit.textui.TestRunner.run(suite());junit.textui.TestRunner.run(suite());Example main class for running testsExample main class for running tests// imports// importspublic class ATestSuite {public class ATestSuite {

public static TestSuite suite () { public static TestSuite suite () { TestSuite suite = new TestSuite(“MyPackageTests”); TestSuite suite = new TestSuite(“MyPackageTests”); suite.addTestSuite(“MyClassTests.class”); suite.addTestSuite(“MyClassTests.class”); return suite; return suite;

}} public static void main(String args[]) { public static void main(String args[]) {

junit.textui.TestRunner.run(suite());junit.textui.TestRunner.run(suite()); }}}}

Assert methodsAssert methods

There are a variety of assert methods There are a variety of assert methods available, e.g.available, e.g. AssertTrue, assertFalse, AssertNull, AssertTrue, assertFalse, AssertNull,

assertSame, etcassertSame, etc

In general, the syntax is In general, the syntax is assert<Foo>(<type> expected, <type> assert<Foo>(<type> expected, <type>

testVal)testVal) Assert<Foo>(String message, <type> Assert<Foo>(String message, <type>

expected, <type> testVal)expected, <type> testVal)

Project structureProject structure

One way to organize tests is to put them One way to organize tests is to put them in a parallel directory treein a parallel directory tree E.g. MyPackage. …. Has a parallel tree of E.g. MyPackage. …. Has a parallel tree of

tests.MyPackage. ….tests.MyPackage. ….

Test suites can be collected, so Test suites can be collected, so each class has a test classeach class has a test class each directory has test suite with a runner, each directory has test suite with a runner,

which runs the tests in that directory and which runs the tests in that directory and the suites in all subdirectoriesthe suites in all subdirectories

Junit addonsJunit addonsJunit has a bunch of add on stuff that has been written Junit has a bunch of add on stuff that has been written for it (your mileage may vary)for it (your mileage may vary)ExamplesExamples Code generators – make guesses at what tests are Code generators – make guesses at what tests are

neededneeded Load testers Load testers Code coverageCode coverage Testers for thread safetyTesters for thread safety ‘‘Helpful classes’ (e.g. Helpful classes’ (e.g. RecursiveTestSuiteRecursiveTestSuite is a is a

TestSuite that will recursively walk through the TestSuite that will recursively walk through the directory structure looking for subclasses of directory structure looking for subclasses of TestCase and will add them.)TestCase and will add them.)

Cactus, HttpUnit, Mock ObjectsCactus, HttpUnit, Mock Objects

Cactus (from jakarta) is a simple test Cactus (from jakarta) is a simple test framework for unit testing server-side java framework for unit testing server-side java code (Servlets, EJBs, Tag Libs, Filters, ...). code (Servlets, EJBs, Tag Libs, Filters, ...). HttpUnit emulates the relevant portions of HttpUnit emulates the relevant portions of browser behavior to allow automated testing browser behavior to allow automated testing (on sourceforge)(on sourceforge)Mock objects emulate objects from other Mock objects emulate objects from other parts of the application, but have controlled parts of the application, but have controlled behaviourbehaviour

ResourcesResources

VideosVideos http://video.google.com/videosearch?http://video.google.com/videosearch?

q=techtalks&page=1&lv=0&so=1q=techtalks&page=1&lv=0&so=1

Web sitesWeb sites Junit - Junit - http://www.junit.orghttp://www.junit.org CppUnit - CppUnit - http://cppunit.sourceforge.net/http://cppunit.sourceforge.net/ Ant - Ant - http://jakarta.apache.org/ant/index.htmlhttp://jakarta.apache.org/ant/index.html

BookBook Java Tools for Extreme Programming: Mastering Open Java Tools for Extreme Programming: Mastering Open

Source Tools Including Ant, JUnit, and CactusSource Tools Including Ant, JUnit, and Cactus