Effective Unit Testing with Effective Unit Testing with NUnitNUnit

Win-Dev 2004 Win-Dev 2004 Friday, October 29, 2004Friday, October 29, 2004

Peter Provost Peter Provost Software Design EngineerSoftware Design Engineerpatterns & practicespatterns & practicesMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

What is Unit Testing?What is Unit Testing?

Typical Unit Testing ProblemsTypical Unit Testing Problems

Best Practices for Effective Unit Best Practices for Effective Unit TestingTesting

Tools DemoTools Demo

Q & AQ & A

What is Unit Testing?What is Unit Testing?

A DefinitionA Definition

Unit Testing is Unit Testing is code that…code that…

Is written by Is written by developers, for developers, for developers.developers.

Exercises a small, Exercises a small, specific area of specific area of functionality.functionality.

Helps “prove” that Helps “prove” that a piece of code a piece of code does what the does what the developer expects developer expects it to do.it to do.

What Unit Testing Is NotWhat Unit Testing Is Not

Acceptance Testing (aka Functional Acceptance Testing (aka Functional Testing)Testing)

Performance TestingPerformance Testing

Scalability TestingScalability Testing

Marick’s Four Quadrants of Marick’s Four Quadrants of TestingTesting

Source: Brian Marick – http://testing.com/

CustomerTests

Programmer Tests

Exploratory Tests

ScalabilityPerformance

UsabilitySecurity

Business Facing

Technology Facing

Sup

port

Dev

elop

men

tC

ritique Result

Why Unit TestWhy Unit Test

It will make your It will make your life easierlife easier

Better codeBetter code

Better designsBetter designs

Code is easier to Code is easier to maintain latermaintain later

Confidence when Confidence when you codeyou code

Common ExcusesCommon Excuses

I’m not a tester!I’m not a tester!

It takes too much It takes too much time.time.

It takes too long to It takes too long to run the tests.run the tests.

I don’t know how I don’t know how to test it.to test it.

I don’t really know I don’t really know what it is supposed what it is supposed to do, so I can’t to do, so I can’t test it.test it.

But it compiles! It But it compiles! It doesn’t need tests.doesn’t need tests.

Typical Unit Testing Typical Unit Testing ProblemsProblems

Testing Is Monotonous and Testing Is Monotonous and BoringBoring

Often indicates bad tooling or bad Often indicates bad tooling or bad techniquestechniques

Unit testing should…Unit testing should…Be easy to doBe easy to do

Provide rapid feedbackProvide rapid feedback

Poor Test CoveragePoor Test Coverage

Often happens when tests not written Often happens when tests not written firstfirst

Hard to retrofit laterHard to retrofit later

Test coverage is orthogonal to test Test coverage is orthogonal to test qualityquality

Purpose of Tests is Purpose of Tests is MisunderstoodMisunderstood

Remember why we do unit testingRemember why we do unit testing

Easy to get distracted by irrelevant Easy to get distracted by irrelevant thingsthings

What is the scope of the test?What is the scope of the test?

Informal Testing ProcessInformal Testing Process

How does your team do unit testing?How does your team do unit testing?

Test-firstTest-first

Intuitive “poking and prodding” style Intuitive “poking and prodding” style testingtesting

Smoke testingSmoke testing

Whatever the developer wantsWhatever the developer wants

Inconsistent TestingInconsistent Testing

Are tests required before check in?Are tests required before check in?

How is it enforced?How is it enforced?

Low Test QualityLow Test Quality

Single biggest problem in unit testing Single biggest problem in unit testing todaytoday

Testing requires a special mindsetTesting requires a special mindset

Tools only solve half the problemTools only solve half the problem

Code Not Designed for TestCode Not Designed for Test

Often the cause of low test qualityOften the cause of low test quality

Hard to design testable codeHard to design testable code

However, testable code often is However, testable code often is better designedbetter designed

Tests Not MaintainedTests Not Maintained

So you’ve shipped your code and So you’ve shipped your code and someone finds a bug…someone finds a bug…

Do you fix the tests? Or just fix the Do you fix the tests? Or just fix the bug?bug?

What about the rest of the team?What about the rest of the team?

Best Practices for Effective Best Practices for Effective Unit TestingUnit Testing

Automate Your TestsAutomate Your Tests

If it is hard to run tests, you won’t do If it is hard to run tests, you won’t do itit

Manual (aka scripted) tests make Manual (aka scripted) tests make regression testing very hardregression testing very hard

Use Good ToolsUse Good Tools

The tools should make it…The tools should make it…Easy to write testsEasy to write tests

Easy to organize your testsEasy to organize your tests

Easy to run tests (all, some, one)Easy to run tests (all, some, one)

They should provide…They should provide…Quick feedback of pass/failQuick feedback of pass/fail

Details on the fail conditionsDetails on the fail conditions

Use Good ToolsUse Good Tools

Testing Focused Testing Focused ToolsTools

NUnitNUnit

csUnitcsUnit

MbUnitMbUnit

Test Driven .NET Test Driven .NET (aka NUnitAddin)(aka NUnitAddin)

NMockNMock

DotNetMockDotNetMock

Supporting ToolsSupporting Tools

NAntNAnt

CodeRushCodeRush

ResharperResharper

CodeSmithCodeSmith

And of course…And of course…

Visual Studio .NETVisual Studio .NET

Tools DemoTools Demo

Get a MentorGet a Mentor

Testing is as much an art as a Testing is as much an art as a sciencescience

Learning to write good tests is hardLearning to write good tests is hard

Some people are naturally good at it Some people are naturally good at it and some aren’tand some aren’t

Use a Test ListUse a Test List

Start your development activities by Start your development activities by writing down a list of things you want writing down a list of things you want to testto test

You will often think of a test while You will often think of a test while writing another one. When you do, writing another one. When you do, add it to the list.add it to the list.

Review your list frequentlyReview your list frequently

Write Tests FirstWrite Tests First

Test-driven development (TDD) is a Test-driven development (TDD) is a proven way of improving quality*proven way of improving quality*

TDD’s main objective is not testing TDD’s main objective is not testing software! (this is a side effect)software! (this is a side effect)

TDD’s main objective is to aid TDD’s main objective is to aid programmers and customers during programmers and customers during the development process with the development process with unambiguous requirements. unambiguous requirements.

Code is written with testing as a Code is written with testing as a primary motivation. In short, the primary motivation. In short, the code is testable.code is testable.

* - http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf* - http://collaboration.csc.ncsu.edu/laurie/Papers/TDDpaperv8.pdf

What to TestWhat to Test

The “did we get what we expected” The “did we get what we expected” test isn’t always good enoughtest isn’t always good enough

Ask yourself the question…Ask yourself the question…

““If the code ran correctly, how would I If the code ran correctly, how would I know?”know?”

What to Test - Testing What to Test - Testing HeuristicsHeuristics

Test at the boundariesTest at the boundaries

Test every error messageTest every error message

Test different configurationsTest different configurations

Run tests that are annoying to setupRun tests that are annoying to setup

Avoid redundant testsAvoid redundant tests

Source: Lessons Learned in Software Testing, by Cem Kaner, James Source: Lessons Learned in Software Testing, by Cem Kaner, James Bach, Brett PettichordBach, Brett Pettichord

What to Test – BICEPWhat to Test – BICEP

BB Are the boundary conditions Are the boundary conditions correct?correct?

II Can you check inverse Can you check inverse relationships?relationships?

CC Can you cross check using otherCan you cross check using othermeans?means?

EE Can you force an error condition toCan you force an error condition tohappen?happen?

PP Are the performance characteristicsAre the performance characteristicsacceptable?acceptable?

Source: Pragmatic Unit Testing in C# with NUnit by Andrew Source: Pragmatic Unit Testing in C# with NUnit by Andrew Hunt and Dave ThomasHunt and Dave Thomas

Test StructureTest Structure

William Wake’s 3-A PatternWilliam Wake’s 3-A PatternArrangeArrange

ActAct

AssertAssert

Designing for TestDesigning for Test

One simple question to help you One simple question to help you write good code…write good code…

““How am I going to test this?”How am I going to test this?”

If you don’t know the answer, then If you don’t know the answer, then you probably need to reconsider your you probably need to reconsider your design.design.

Stubs and MocksStubs and Mocks

Stub DefinedStub Defined“ “ A portion is a portion of code that is A portion is a portion of code that is inserted at runtime in place of real code, in inserted at runtime in place of real code, in order to isolate calling code from the real order to isolate calling code from the real implementation. The intent is to replace a implementation. The intent is to replace a complex behavior with a simpler one that complex behavior with a simpler one that allows independent testing of some portion allows independent testing of some portion of real code.” of real code.”

Mock Objects DefinedMock Objects Defined““A mock object is an object created to A mock object is an object created to stand in for an object that your code will stand in for an object that your code will be collaborating with. Your code can call be collaborating with. Your code can call methods on the mock object, will deliver methods on the mock object, will deliver results as set up by your tests.” results as set up by your tests.”

Source: JUnit in Action, by Vincent MassolSource: JUnit in Action, by Vincent Massol

When are stubs appropriate? When are stubs appropriate?

Objects that are expensive to create Objects that are expensive to create to manipulateto manipulate

At the edges of a system, or around At the edges of a system, or around complex clusters of objectscomplex clusters of objects

Course-grained testing, integration Course-grained testing, integration testingtesting

Stub DownsidesStub Downsides

A Complex system might require a A Complex system might require a complex stubcomplex stub

Hard to write in a verifiable wayHard to write in a verifiable way

Difficult to maintainDifficult to maintain

When are mocks appropriate? When are mocks appropriate?

Real object has Real object has non-deterministic non-deterministic behaviorbehavior

Real object is Real object is difficult to set up difficult to set up

Real object has Real object has behavior that is behavior that is difficult to causedifficult to cause

Real object is slowReal object is slow

Source:http://c2.com/cgi/wiki?Source:http://c2.com/cgi/wiki?MockObjectsMockObjects

Mock DownsidesMock Downsides

Code complexity of being able to Code complexity of being able to switch between real and mock switch between real and mock implementationsimplementations

Maintaining the mockMaintaining the mock

Properties of Good TestsProperties of Good Tests

AutomaticAutomaticTests need to be run checkedTests need to be run checkedautomaticallyautomatically

ThoroughThorough Test everything that couldTest everything that couldpossibly breakpossibly break

RepeatableRepeatable Tests should produce the Tests should produce the samesame

results each time they are results each time they are runrunIndependentIndependent A test should exercise only A test should exercise only one one

thing at a timething at a timeProfessionalProfessional Tests are code too! Write Tests are code too! Write themthem

professionally.professionally.

Source: Pragmatic Unit Testing in C# with NUnit by Andrew Source: Pragmatic Unit Testing in C# with NUnit by Andrew Hunt and Dave ThomasHunt and Dave Thomas

Test Organization OptionsTest Organization Options

Tests in the VS Project being testedTests in the VS Project being testedBest for experiments, spikes, etc.Best for experiments, spikes, etc.

Tests in a separate VS ProjectTests in a separate VS ProjectMost commonly used arrangementMost commonly used arrangement

Tests are not part of distributionTests are not part of distribution

Tests in a separate VS SolutionTests in a separate VS SolutionThe usual reference issues, but possibleThe usual reference issues, but possible

Recommended Test Recommended Test OrganizationOrganization

Tests in a separate project/assemblyTests in a separate project/assembly

Naming conventionsNaming conventions

If you are testing the class:If you are testing the class:

Foo.Bar.BazFoo.Bar.Baz

Then you should name your testsThen you should name your tests

Foo.Bar.Tests.BazTestsFoo.Bar.Tests.BazTests

Note the namespace and test class Note the namespace and test class name!name!

When to Run TestsWhen to Run Tests

When you write a new method…When you write a new method………compile and run local unit testscompile and run local unit tests

When you fix a bug…When you fix a bug………run the test that illustrates that bug.run the test that illustrates that bug.

Any successful compile…Any successful compile………run local unit tests.run local unit tests.

Before you check in…Before you check in………run all tests.run all tests.

Continuously…Continuously…...check out and build the project from ...check out and build the project from

scratch including all unit tests.scratch including all unit tests.

Do not check in code that…Do not check in code that…

Is incomplete (e.g. Is incomplete (e.g. missing missing dependencies)dependencies)

Doesn’t compileDoesn’t compile

Compiles but Compiles but breaks other codebreaks other code

Doesn’t have unit Doesn’t have unit teststests

Has failing unit Has failing unit teststests

Passes its tests but Passes its tests but causes other tests causes other tests to fail.to fail.

Additional ResourcesAdditional Resources

Web SitesWeb Sites

http://www.testdriven.comhttp://www.testdriven.com

http://www.xprogramming.comhttp://www.xprogramming.com

http://workspaces.gotdotnet.com/tddhttp://workspaces.gotdotnet.com/tdd

WeblogsWeblogs

Peter ProvostPeter Provosthttp://www.peterprovost.orghttp://www.peterprovost.org

Jim NewkirkJim Newkirkhttp://weblogs.asp.net/jamesnewkirkhttp://weblogs.asp.net/jamesnewkirk

Brian ButtonBrian Buttonhttp://dotnetjunkies.com/WebLog/oneagihttp://dotnetjunkies.com/WebLog/oneagilecoder/lecoder/

Martin FowlerMartin Fowlerhttp://www.martinfowler.com/bliki/http://www.martinfowler.com/bliki/

Darrell NortonDarrell Nortonhttp://dotnetjunkies.com/WebLog/darrell.http://dotnetjunkies.com/WebLog/darrell.norton/norton/

Mailing ListsMailing Lists

Yahoo Group: testdrivendevelopmentYahoo Group: testdrivendevelopment

Yahoo Group: agiledotnetYahoo Group: agiledotnet

Yahoo Group: extremeprogrammingYahoo Group: extremeprogramming

BooksBooks

Test-Driven Development in Microsoft Test-Driven Development in Microsoft .NET – James Newkirk and Alexei .NET – James Newkirk and Alexei VorontsovVorontsov

Pragmatic Unit Testing in C# with Pragmatic Unit Testing in C# with NUnit – Andrew Hunt and David NUnit – Andrew Hunt and David ThomasThomas

Test-Driven Development, by Test-Driven Development, by Example – Kent BeckExample – Kent Beck

Refactoring to Patterns – Joshua Refactoring to Patterns – Joshua KerievskyKerievsky

Lessons Learned in Software Testing Lessons Learned in Software Testing – Cem Kaner, James Bach, and Brett – Cem Kaner, James Bach, and Brett PettichordPettichord

Questions? CommentsQuestions? Comments

Peter Provost Peter Provost Software Design EngineerSoftware Design Engineerpatterns & practicespatterns & practicesMicrosoft CorporationMicrosoft Corporation

E-mail: peterpr@microsoft.comE-mail: peterpr@microsoft.comWeblog: Weblog: http://www.peterprovost.org/http://www.peterprovost.org/

NOTENOTE: Updated slides available at : Updated slides available at http://www.peterprovost.org/Files/C1http://www.peterprovost.org/Files/C16_Effective_Unit_Testing.ppt6_Effective_Unit_Testing.ppt