james grenning test is not about finding bugs

[email protected]

Copyright ©2008-2009 James W. GrenningAll Rights Reserved. . Photos from James Grenning’s Personal Collection or Wiki-Commons

Test is NOT About Finding Bugs!

James W GrenningRenaissance Software Consulting, Co.

[email protected]: jwgrenning

1

1Monday, October 18, 2010

[email protected]

Copyright ©2008-2010 James W. GrenningAll Rights Reserved.

2


[email protected]


www.agilemanifesto.org

3


[email protected]


4


[email protected]


More Functionality in Software

6 years

5


[email protected]


More and More Software

6


[email protected]


7


[email protected]


8


[email protected]


9


[email protected]


10


[email protected]


11


[email protected]


12


[email protected]


13


[email protected]


14


[email protected]


15


[email protected]


16


2000’s

[email protected]


Now vs. Then

17

SW

SW

HW

1960’s

SW

HW


[email protected]


The Nature of Software

• Software is fragile– The nature of discrete systems

• Any change can break just about anything• Test and forget model leads to big surprises and

problems

18


[email protected]


19


[email protected]


Lot’s of Software meansLot’s of Problems

• 10-50 defects per 1000 Lines of Code (KLOC)

• How many bugs in

20


[email protected]


21


[email protected]


Voyager

22


[email protected]


23


[email protected]


Mars Climate (burns in) Orbiter

24


[email protected]


25


[email protected]


1990 - AT&T #4ESS

26


[email protected]


27


[email protected]


28


[email protected]


29

Did you say something?

What the heck!?! Where are we?


[email protected]


30


[email protected]


Defect Found During Simulation

31


[email protected]


32


[email protected]


33


[email protected]


34


[email protected]


Don’t Test to Find DefectsTest to Prevent Defects

• Automate• Test as you go• Re-test everything

• Code is guilty until proven innocent

35


[email protected]


Manual Test Is Unsustainable

36


[email protected]


Et = f(Ed)Et is the effort to test a new feature, and is a function of the effort to develop the feature.

Ed is the effort to develop a new featureAssume a constant linear relationship

Et = f(Ed) = KEd

37


[email protected]


Conservative Estimate of Developer to Tester Ratio

38

0

5

10

15

1 2 3 4 5 6 7 8 9 10

Num

ber

of P

eopl

e

Iteration

New functionality manual test effort

New functionality development effort


[email protected]


It’s not that EasySoftware is Fragile

• 25% of defects are introduced while changing existing code (study by Grady)– The new code works– Some existing code is unintentionally broken

39


[email protected]


Etn = f(Ed) + C Et(n)Etn is the effort to fully test a product at iteration N

Because any change can break previously working software, we must retest.

Etn is a function of the effort to develop the feature plus some fraction of the effort to test all previous iterations.

40

i = 0

n

x


[email protected]


Unsustainable Growth with any Coefficient

41

0

5

10

15

1 2 3 4 5 6 7 8 9 10

Peop

le N

eede

d

Iteration

Assume regression test effort is 50% original manual test effort

Total manual test effort is unsustainable

Ever Growing Need

Flat Budget

Unt

este

d C

ode

Gap


[email protected]


So, Maybe we should only test at the end of the development cycle

42


[email protected]


FBI Case-File System

43


[email protected]


Cost of BDUF gone bad

44


[email protected]


Should We Wait Until the End to Test?

Requirements

Code

Design

Test

Time

NovemberSeptemberJulyMayMarch

Test and Fix

Test and Fix

Test and Fix

Test and FixTest and Fix

Test and Fix

Test and Fix

Test and Fix

Test and Fix


Test and Fix

Test and Fix



45

Test


[email protected]


This Work Flow is Designed to Allow Defects

46

Development

Test

Defects


[email protected]


Development and Test Work TogetherPreventing Defects

47

Development

Test


[email protected]


Testing is not a Phase

• Testing starts on day one• Tests provide the

specification of what is to be developed

• QA/System Test moves upstream.

48

HELP!


[email protected]


Edsger Dijkstra

Those who want really reliable software will discover that they must find means of avoiding the majority of bugs to start with, and as a result, the programming process will become cheaper. If you want more effective programmers, you will discover that they should not waste their time debugging, they should not introduce the bugs to start with.

49


[email protected]


Can we Realize Dijkstra’s Dream andPrevent Defects with

Test Driven Development?

50


[email protected]


Zune 30G

• On December 31, 2008 this function went into an infinite loop.

• Ruining New Years Eve Parties around the world.

51


The buggy function:

It accepts the number of days since January 1, 1980 and figures out: day, date month and year.

[email protected]


52

Zune 30G Bug

BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)


[email protected]


53

This Test Could Have Prevented the Zune Bug

TEST(Rtc, check20081231){ days = daysSince1980(2008, 366); CHECK(ConvertDays(days, &time)); assertDate(WED, 31, 12, 2008);}


[email protected]


What Tests do we Need?

• One Test Would Have Prevented Zune Bug• If we knew where the bugs are, we could just write

those tests (or fix the bug).• Bugs can be anywhere, so we have to write tests for

everything.

54


[email protected]


Uncle Bob’s Three Rules of TDD

1.You are not allowed to write any production code unless it is to make a failing unit test pass.

2.You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.

3.You are not allowed to write any more production code than is sufficient to pass the one failing unit test.

source: Robert C. Martin, blog.objectmentor.com

55


[email protected]


TDD State Machine

56

Write the test

Deal with the linker

Deal with the Compiler

Compilation error

Link error

New test failsMake it work

Refactor(Make it right)

All tests pass

All tests pass No more tests

Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


The Physics of Debug Later Programming (DLP)

• As Td increases, Tfind increases dramatically• Tfix is usually short, but can increase with Td

57

Bug discoveryMistake made(bug injection)

Bug found Bug fixed

Td Tfind T fix

Time


[email protected]


The Physics of Test Driven Development

• When Td approaches zero, Tfind approaches zero• In many cases, bugs are not around long enough to be considered

bugs.• See: http://www.renaissancesoftware.net/blog/archives/16

58

Mistake discovery

Mistake made

Root cause found

Mistake fixed

T d Tfind T fix

Time


[email protected]


TDD Progress in the Right Direction

• See: http://www.renaissancesoftware.net/blog/archives/33

59

Code

without

new feature

Code with

new untested

feature

Code with

new tested

feature

Write all production code at once

Debug

Test Driven Development

Write the test

Deal with the linker

Deal with the Compiler

Compilation error

Link error

New test failsMake it work

Refactor(Make it right)

All tests pass

All tests pass No more tests

Choose a test

Start

Compilation error

Link error

DONE!

Programming error

Compiles Clean


[email protected]


How Can we Keep the Cost or Retest Sustainable? -- Automate

60

0

5

10

15

1 2 3 4 5 6 7 8 9 10

Peop

le N

eede

d

Iteration

Assume automation test effort is 100% more than original manual test effort

Relatively Flat and Sustainable


[email protected]


How Can we Keep the Cost or Retest Sustainable? -- Automate

61

0

5

10

15

1 2 3 4 5 6 7 8 9 10

Peop

le N

eede

d

Iteration

Relatively Flat and Sustainable

Ever Growing Need

Flat Budget

Unt

este

d C

ode

Gap


[email protected]


But TDD won’t work for us!We have special problems!

• Fire alarm• Home automation• Banking• Insurance• Robotics• Medical electronics• Printing• Gambling machines

• Telecom• Police radio• Cell phone• On-line gaming• Security software• Mesh networks• Test equipment• Medical records

62


[email protected]


Yes, You are Special and Unique

• But TDD can work in many environments

63


[email protected]


But I’ve seen it work.Everywhere someone lets it.

• Fire alarm• Home automation• Banking• Insurance• Robotics• Medical electronics• Printing• Gambling machines

• Telecom• Police radio• Cell phone• On-line gaming• Security software• Mesh networks• Test equipment• Medical records

64


[email protected]


But I’m special

I have ____________ Dependencies

65

Fill in the blank*

*Hardware, OS, Database, Regulatory, Real-time, User, Network...


[email protected]


Unmanaged Hardware Dependency Lead to Manual Testing

66

The Net

Core Software


[email protected]


But I Have Hardware Dependencies

• Break the dependencies

67

<<interface>>

Time Service

+ getDay()+ getTimeOfDay()+ setPeriodicAlarm()

LightScheduler

Test

LightScheduler

+addSchedule()+removeSchedule()+wakeUp()

<<interface>>

Light Controller

+ on(id)+ off(id)

Light Controller Spy

FakeTime Service

<<implements>> <<implements>>


[email protected]


Design Must Accommodate Test

TEST(LightScheduler, TurnOnEverydayNotTimeYet) { LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510); whenItBecomes(TimeService::MONDAY, 509); expectLight(3, LIGHT_UNCHANGED); }

TEST(LightScheduler, TurnOnEveryday) { LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510); whenItBecomes(TimeService::MONDAY, 510); expectLight(3, LIGHT_ON); }

68


[email protected]


Managed Dependencies Enables Automated Testing

69

Copyright (c) 2009 James Grenning

Core Software

Test scripts

Test Agent

Test Controller


[email protected]


There are Different kinds of Automated Tests

• Unit Test – Feedback to the developer that the code does what is expected– Written using a Unit Test Harness (e.g. unity, CppUTest, …)

• Story Tests - Executable use cases - Our focus– Feedback to the Customer that the code meets the requirements– Used at many levels

– Component– Groups of integrated components– System

– Written in a domain specific language (e.g. FitNesse) [FITNESSE]

• Integration, System, Load Tests...

70


[email protected]


Unit tests are critical

• A house made from solid bricks can stand a long time• TDD gives solid building blocks• Unit tests tell programmers that code works like they

think it works

71


[email protected]


System Level Tests Cannot be Thorough

• 1000 (or more) tests are needed to test this simple system

72

10 states

10 states

10 states

5 interactions

5 interactions

5 interactions

Tests


[email protected]


TDD unit tests can be thorough

• As few as 30 unit tests and 15 integration test when tested as units

73

10 states

10 states

10 states

5 interactions

5 interactions

5 interactions

Tests Tests


[email protected]


Higher Level Tests are Needed Too

Could Test Help Specify what is needed with Executable Use cases

See http://renaissancesoftware.net/papers.htmlFor Executable Use Cases Paper

74


[email protected]


Use-Case Template

• Name:• Goal: • Preconditions:• Success End Condition: • Failed End Condition: • Primary Actor:• Trigger:• MAIN SUCCESS SCENARIO• EXTENSIONS

Source Alistair Cockburn http://alistair.cockburn.us/Basic+use+case+template

75


[email protected]


Use-Case Example

76

Information Description

Name Schedule light control Goal Allow system users to schedule lights to turn on, off, or

dimPreconditions System has controllable lights attachedSuccess End Condition The scheduled light has been controlled at the scheduled

timeFailed End Condition The scheduled light has not been controlled at the

scheduled timePrimary Actor Home ownerTrigger Scheduled time is reachedMain Success Scenario 1.The home owner schedules a light to turn on at a

specific time on a specific day

2.The scheduler wakes up at the right time of the right day

3.The light scheduled for this minute is turned on


[email protected]


Use-Case ExampleContinued

77

Information Description

Extensions/Variations 1a. Homeowner can schedule the light to turn on

1b. Homeowner can schedule the light to turn off

1c. Homeowner can schedule the light to set to a dim

level

1d. Homeowner can specify weekend schedule

1e. Homeowner can specify weekday schedule

2a - Scheduler does nothing when it wakes up and there

are no scheduled controls.

3a - Light is turned on when on is scheduled

3b - Light is turned off when off is scheduled

3c - Light is set to a specified level when dim is scheduled


[email protected]


Executable Use-Case with FitNesse

78

Items in tables are tell to FitNesse how to interact with the system under test

Initialization

Cleanup


[email protected]


Passing Test

79


[email protected]


Failing Test

80


[email protected]


A Suite of Tests While all Test Pass

81


[email protected]


When the Story Test is Ready Before the Development Finishes

82


[email protected]


When Things are not Working

83


[email protected]


FitNesse Test Architecture

84

<<interface>>

Time Service

+ GetTime()+ SetPeriodicAlarm()

LightScheduler

Fixture

LightScheduler

+ ScheduleTurnOn()+ RemoveSchedule()+wakeUp()

<<interface>>

Light Controller

+ On(id)+ Off(id)

Light Controller Spy

FakeTime Service

<<implements>> <<implements>>

FitNesseWiki Server

CSlim Library

<<socket>>FitNesse Test Pages


[email protected]


Test is NOT About Finding Bugs!

• TDD to prevent bugs• Automate repetitious tests

– to keep bugs out– to keep the cost of re-test low

• It’s not easy, until you learn how to do it– then it’s still not always easy, but worth it

85


[email protected]


Don’t let your Product Appear on a Future Famous Failures Presentation

86


[email protected]


Thank You!

87


[email protected]


Available now in Beta

http://pragprog.com/titles/jgade/

In print early 2011

88


james grenning test is not about finding bugs

Documents