testing: it’s part of your job — make it effective!egu/eto2007/testy.pdf · challenges of...

Copyright is held by the author/owner(s). OOPSLA’07 October 21–25, 2007, Montréal, Québec, Canada. ACM 07/0010.

ooPSLA is sponsored by ACM SIGPLAN in cooperation with SIGSOFT

22ND INTERNATIONAL CONFERENCE ON OBJECT-ORIENTED PROGRAMMING, SYSTEMS, LANGUAGES, AND APPLICATIONS

Testing: It’s Part of Your Job — Make it Effective!

Neil B. Harrison Utah Valley State College

Orem, Utah, USA [email protected]

The Association for Computing Machinery 2 Penn Plaza, Suite 701 New York, New York 10121-0701 Copyright © 2007 by the Association for Computing Machinery, Inc. (ACM).

This compilation is copyright by ACM, however copyright for the individual papers is owned by the author/owner(s). Copying for personal or classroom use is permitted. To copy otherwise, to republish, to post on servers or redistribute to lists, requests specific permission from the authors.

Notice to Past Authors of ACM-Published Articles ACM intends to create a complete electronic archive of all articles and/or other material previously published by ACM. If you have written a work that has been previously published by ACM in any journal or conference proceedings prior to 1978, or any SIG Newsletter at any time, and you do NOT want this work to appear in the ACM Digital Library, please inform [email protected], stating the title of the work, the author(s), and where and when published. Printed in the USA

OOSLA 2007 Tutorial iii Neil Harrison

Table of Contents

1. Introduction

a. Importance of Testing

b. Challenges of Testing

c. Purposes of Testing

2. The Nature of Testing

a. Black Box and White Box

b. Developers and Testers

c. Testing and Agile Development

3. Developer Mind Games

a. Principles to Remember

b. Mindset Benders

4. Testing Smart

a. Testing Smart Automatically

b. Test Design Techniques

i. Equivalence Classes

ii. Boundary Values

5. Putting the Test into TDD

6. Summary

a. Final Exam

Testing: It's Part of Your Job - Make it Effective!

OOPSLA 2007 Tutorial iii Neil B. Harrison

OOSLA 2007 Tutorial iv Neil Harrison

Instructor Biography

Neil Harrison is an assistant professor of computer science at Utah Valley State College in Orem, Utah. Before that, he was a distinguished member of technical staff at Avaya Labs, where he developed communications software led software development teams. He has studied software development organizations for over ten years and is a co-author of Organizational Patterns of Agile Software Development. He has published numerous articles on software patterns, effective organizations, and software testing. He is acknowledged as the world's leading expert on pattern shepherding and the pattern conference (PLoP) shepherding award is named after him.

Professor Harrison has a BS in Computer Science with High Honors and University Scholar Designation from Brigham Young University, and an MS in Computer Science from Purdue University. He is a member of ACM, The Hillside Group Board of Directors, and Hillside Europe.


OOPSLA 2007 Tutorial iv Neil B. Harrison

OOSLA 2007 Tutorial v Neil Harrison

Agenda Part 1: (90 minutes): Units 1-3 Break (30 minues)

Part 2: (90 minutes): Units 4-6, and final exam


OOPSLA 2007 Tutorial v Neil B. Harrison

OOSLA 2007 Tutorial vi Neil Harrison

Tutorial Objectives You should learn the following:

1. The difficulties of testing software, especially for developers 2. The particular difficulties that developers have testing their own code 3. The purposes of software testing 4. Challenges of testing in agile development, including pitfalls of TDD 5. Principles and mindset benders for testing 6. Test design techniques

You should be able to:

1. Design a testing approach (for developers) 2. Write systematic unit tests following proven techniques 3. Approach testing, including TDD, in a systematic way, resulting in more effective

testing


OOPSLA 2007 Tutorial vi Neil B. Harrison

OOSLA 2007 Tutorial vii Neil Harrison

Introduction For developers, testing is hard. Even worse, it can be remarkably ineffective: we often invest a great deal of time and effort in test development and execution, only to let significant bugs get through to the customer. Developers are generally poor testers, but developers MUST test – they can’t simply push all testing off to system or acceptance testers. Fortunately, there are things you can do to make your development testing much more effective. The answer lies in the way you approach writing your tests. Through a combination of objective and subjective techniques, you can write tests that are more likely to uncover bugs in your software. The best part is that it doesn’t take a lot of effort – you will be working smarter, not harder. In addition, the basics of these techniques are easily learned. In this tutorial, you will learn how to write tests that give better coverage, find more errors, and are simply more effective for the time invested. You will also learn key mindsets that will make you more aware of where bugs may lurk. You will learn how to integrate these techniques with test-driven development (TDD), which makes TDD not only an important development tool, but a powerful quality assurance tool at the same time. Prior knowledge of TDD is not required.


OOPSLA 2007 Tutorial vii Neil B. Harrison

1

Testing: Its Part of Your Job –Make It Effective!

Neil HarrisonUtah Valley State College

[email protected]

Readme First!

This course contains multiple exercisesThe answers follow each slide

Don’t look – it will spoil the learning experienceWe will go over the answers in class


OOPSLA 2007 Tutorial 1 Neil B. Harrison

2

Unit 1: Introduction

Testing:How Hard Can it Be?

Why Is Testing Important?

Because:Software quality is importantBut it is impossible to write perfect code Testing is a known way to help improve software quality



3

Exercise 1.1

Write test cases to adequately test the following program:

The program takes 3 integer inputs, representing the lengths of the sides of a triangle.The program reports whether the triangle is scalene, isosceles, or equilateral.

(Your answers here)



4

Exercise 1.1 answers (1):

A test for an equilateral triangleA test for a scalene triangleA test for an isosceles triangleAll permutations of an isosceles triangleOne side == sum of the other twoAll permutations of such numbersOne side > sum of the other twoAll permutations of such numbers

Exercise 1.1 answers (2):

A negative numberAll permutations of a negative numberAll zeros in the inputWrong type of input (e.g. characters)Too many inputsToo few inputs



5

Exercise 1.1 answers (3)

For Each Test:Did you specify the expected result?

Why is this important?

Exercise 1.1 Scoring

One point for each bullet

Highly experienced developers scored 7.4 out of 14!

Um, which 6.6 test cases would you omit?



6

Source and Caveat

Glenford Myers: “The Art of Software Testing” 1979

Caveat: I can think of at least one important test case he missed – can you?

We will cover it later

Exercise 1.2Testing Software is Hard

Why is it hard? (your answers here)



7

Why is Testing Software Hard?

So many test cases neededThis is the big reason!

You might get the test cases wrongThe results may change as the execution context changesIt’s hard to think of things to test (effectively)

This is the second big reason!

Number of test cases

How many test cases needed to exhaustively test the triangle program?



8

Number of test cases

How many test cases needed to exhaustively test the triangle program?

All valid inputs: triplets of all positive integersAll invalid inputs: everything else

For all but the most trivial program, the number of possible test cases is infinite

Exercise 1.3The Purpose of Testing

What is the purpose of software testing? (your answer here)



9

To prove that the software works?

You can’t do it.Why not?

“Testing cannot prove the absence of bugs, it can only prove the presence of bugs.”

Edsgar Dijkstra

To prove that the software doesn’t work?

Is this very helpful?

What have we learned if we know the software has a bug in it?It may be useful to know where the bugs are



10

To find all the bugs?

Is this possible?

Why not?

To improve product quality?

You can’t test quality into the product.

The output of testing is information, not quality improvement

The information can be used to improve quality



11

To assess the quality of the system?

How is this useful?

Implies thorough testing coverage

To direct quality improvement efforts?

Very Useful – how?

Use bug density to focus efforts



12

To reduce risk of field problems?

Potentially Useful –How?

Test most critical areas in the software

Goals + Reality = Actions

We can’t test everythingWe have limited resourcesSo we must test smart

Test in line with our goalsUse testing methods to get maximum information with the minimum effort.



13

The Hard Thing About Testing

… is deciding what NOT to testJohn Connolly, my first boss

Unit 2

The Nature of Testing



14

Testing topics

In order to be effective testers, we need to understand the nature of testing:

White Box vs. Black BoxDeveloper vs. TesterTesting and Agile Development (including TDD)

Black Box Testing

Based on RequirementsAssumes no knowledge of implementation



15

Black Box Testing:Plusses and Minuses

+ Based on what the user experiences+ Can test the system as a whole+ Can test non-functional aspects

+ Performance, usability, etc.

- Easy to miss things- Like error cases

- Often must wait for the whole system

White Box Testing

Uses knowledge of implementationYou look at the code



16

White Box Testing:Plusses and Minuses+ Can cover all the code

+ and detect unreachable code+ Can be done early

+ during coding+ Easy to measure testing progress- Tests written after code is written

- Biased by the code (HUGE Weakness!!!)- You can’t test code that isn’t there but should

be!

Who are the Testers?

DevelopersTest while code is writtenUnit testing

Dedicated TestersSystem level

You need both!



17

Exercise 2.1Who does what?

Testers

Developers

White BoxBlack Box

Who does what?

NoYesTesters

YesYes (limited)Developers

White BoxBlack Box



18

Note: Tutorial Focus

In my opinionBlack box testing is generally more effective than white box testingThe most effective white box testing technique is a code inspection.

Another technique is McCabe’s Structured Testing

In this tutorial, we will focus on black box testing done by the developer

It’s all in the Mindset

Developers and testers have different mindsetsIt affects the ability to testMore important than people realize



19

Exercise 2.2Fill in the Mindsets

TesterDeveloper

Exercise 2.2Fill in the Mindsets

TesterPessimistDestructiveKnows code is brokenFull coverage is enoughRewarded for quality (we hope!)Knows how to test

DeveloperOptimistConstructiveExpects code to workFew tests is enoughRewarded for writing codeDoesn’t know how to test



20

Developer Limitations

Can’t test the entire systemBiased by the code they writeLack of testing expertisePressure to deliver the code

Developer Advantages

Speedy code-test-fix cycle!

AGILITY!!!

Knows where the bugs are (maybe)Cheaper to fix the bugs



21

Cost of Bugs:Find, Fix, and Impact

0

200

400

600

800

1000

1200

Req. Coding Systest

Field

Cost

The Quandary:

Developers are poor testersBut they must test!

So here are some things to help youHelp your mindsetTesting techniques



22

Unit 2.1

Testing and Agility

Testing in an Agile World

Short SchedulesLimited time for system testing

FlexibilityChanges in feature content

More reliance on developer testing

Is TDD the Answer???



23

TDD ≠ Panacea

Developer testing can help improve qualityBut it cannot replace all testing

TDD is (can be) an approach to developer testing

TDD: A Quick Review

1. Write a test2. Run the tests3. Write code to implement the test(s)4. Repeat steps 2 and 3 until all tests

work.5. Refactor as desired



24

TDD and Testing IssuesCode and tests written by the same mindsetTests are code too!

If a test fails, where is the bug?Test suites can grow to be huge

Execution timeMaintenance time

Refactoring is programmingProgramming introduces bugs by definition

No system viewViolates the Al Capone Principle (later)

Fixing TDD:Make it Systematic

Several things beyond the scope of this class:

ArchitectureDesign

Some things in the scope of the class:Use case driven testing (will touch on it)Testing techniquesMindset shifts



25

Unit 3

Developer Mind GamesTraining Your Brain

Unit 3.1

A Few Principles

Sayings to Remember



26

Principle 1:Mission: Impossible!

Can you ever completely test a system? NO!So don’t trySo select the most effective tests

Other principles and techniques support this

Principle 2:The Insanity Principle

Insanity: doing the same thing and expecting a different resultSoftware: running the same test again “just to be sure”

But we do it!Avoid duplicated tests

with a caveat!



27

Principle 3:The Al Capone Principle

Why did Al Capone rob banks?

Because that’s where the money was!

So run tests where the bugs are most likely to be

Based on experienceBased on analysis

Exercise 3.1Where are the bugs?

In general where are bugs most likely?Your answers here



28

Exercise 3.1Where are the bugs?

New codeBoundariesResource-constrained

Real time constraintsMemory constraints

Highly complexDomain-specific

Example: PBX feature interactions

Principle 4:The Tree in the Forest

If a tree falls in a forest, and nobody hears it, does it make a sound?If there is a bug in the code, and no user finds it, is it a bug?Therefore, focus on failures rather than faults



29

Failures vs. Faults

Fault: a defect in the codeFailure: a mismatch between what the user expects and what the software does

Faults cause failures, but:Not all faults cause failures

Focus on failures (mostly)Tends towards black box testing

Unit 3.2

Mindset Benders



30

Think Like a TesterThese are hints to help you think differently than a typical developerNames designed to be easy to remember

Half the battle is remembering

Patterns printed in “Better Software”

Systematic Sloppy SelfWe have the same blind spots testing code as we do developing it.Therefore, explicitly use a systematic approach

Base tests on use casesOther approaches covered later



31

Use Case Based Testing

Use cases are detailed user scenariosUser Stories are not use cases

DetailsComprehensiveness

Tests must cover use case extensionsThe error legs of use case

Write the tests from the use casesNot from the code that implements a use caseBest: before the code is written

Functionality ChunksWe can’t run every possible testTherefore, divide code into chunks of equivalent functionality

Test each equivalent chunk onceCalled equivalence classesMore detail later



32

Magic Boundaries

Funny things happen at the edgesTherefore, test on both sides of boundaries

Mode details later

Leaky Buckets of MemoryGot memory leaks? How do you know you don’t?

They can even happen in Java!

Therefore, check size of memory before and after extensive testing run

Often system testCan be done at the unit level



33

Triple-X(Execute Exceptions Exhaustively)

Bugs cluster around exceptional behaviorBut it’s easy to forget themTherefore, write tests to cover exceptional behavior

Specify results BEFORE running them

Patterns from the User View

The next patterns are based on the user’s view of the systemThey have more of a system flavor to themTherefore, you might not be able to use them directlyBut even thinking about them will help

More on this point later



34

Creative Idiot Users

Users can do things developers don’t expectTherefore, try things you “know”the users won’t do.

Notes

You normally test the “sunny day”scenariosCreative Idiot Users aims at odd, illegal behaviorIt stress the fault tolerance of your code



35

To Impossibility, and Beyond!

The impossible may be possibleTherefore, consider testing for impossible conditions

You might not test them allApollo 11 story

Notes

Similar to Creative Idiot UsersFocus on impossible conditions, including input from other subsystems or systems

Key Question: What should the code do?

Remember that a part of EVERY test is its expected result!!!



36

You’re Repeating Yourself, You’re Repeating Yourself

Running the code can change its stateIf so, run repetitive tests

Careful: have a good reason for repeating tests!

Notes

This is NOT running a test “just to be sure it works”

Again, specify the expected resultExamples:

Write a record to a DB; then write the same one againChange a user password, then do it again (many systems have password change frequency rules)



37

My Cup Runneth Over

Do you have capacity limits?

DB, users, performance, etc.

If so, test above the limits

Somebody will try itBehavior may “spike”

Notes

Sometimes system level testsBut think about your own piece: what are its limits?

Again, what should the code do in over-capacity situations?

See a recurring theme?



38

Using These Patterns

These are memory joggersRemember the namesOverlap among patterns is intentional

Consider during design, codingBefore coding, spend some time designing tests.

The Test Design Side Benefit

The key question: what should the result of the test be?

Forces thought about desired behavior, especially of unusual casesHelps you think about cases you might not have otherwise considered

This is at design time: you get a more robust design



39

Exercise 3.2

You are writing the inventory management subsystem for the back end of an e-commerce application

1. Note which patterns to use2. Write some tests3. Write requirements questions to be

resolved

Exercise 3.2: Answers:Patterns

Systematic Sloppy SelfParts of use cases

Magic boundariesEspecially when you are out of an item

Triple-ExDB failure, client connection drops

To Impossibility and BeyondIllegal requests, badly formatted requests

You’re Repeating Yourself …Multiple requests for the same item

My Cup Runneth OverMaximum simultaneous requests



40

Exercise 3.2 Answers: Requirements Questions

What should the code do if:The DB is fullThe DB dropsThe client connection dropsYou get a bad requestThe item is out of stockYou have multiple competing threads

Unit 4

Testing Smart



41

Unit 4.1

Testing Smart -- Automatically

Automated Testing

We won’t cover howGet a tool that works for youApply principles learned here

A few PlussesA few MinusesAutomation ≠Panacea



42

Automation Advantages

Fast executionCan automatically record test execution

And results

RepeatableFor developers trying to find bugsRegression testing

Automation DisadvantagesAutomation means you can test badly fasterDoesn’t guarantee good coverage

Even with coverage analysis toolsTest suites grow – a lot!

Can be half or more of the total codeTests are code too

MaintenanceEffortDead Tests

Tests can have bugs too!But we often believe the tests over the codeThe Ada compiler



43

Unit 4.2

Testing Smart:Test Design Techniques

Reminders

We can’t test everythingSo we test intelligently

Where the bugs are most likely hidingWhere the user impact is the greatestWith minimal unneeded duplication

The fewest tests with the greatest coverage



44

Unit 4.2.1

Testing Smart:Equivalence Classes

General Concept

Minimize redundant testsIf the system behaves the same for a given set of inputs, they are all equivalent

So use just one test for them



45

Steps

Identify groups, or classes of input where the expected behavior is the sameWrite one test case for each equivalence class

Don’t forget expected behavior

Exercise 4.1:

A ticketing system:First run movies:

Evening: $9.00Matinee: $6.50

Older movies:Evening: $8.00Passes: $7.00Matinee: $6.00



46

Exercise 4.1 AnswersFirst run movie, evening

Test case: Harry Potter, 19:30, expect $9.00First run, matinee

Similar test casesOlder, eveningOlder, discount passOlder, matineeIllegal/Unrecognized movie typeIllegal movie time

Test case: Harry Potter, 24:30, expect “ERROR”

Exercise 4.2:Country and currency:EU vs. Non-EU

(European Union Countries)EU countries: currency

United Kingdom: the poundDenmark: KronerOther EU countries: the Euro

Non-EU countriesTheir own currency

Countries not recognized by this software



47

Exercise 4.2 answers:EU, UKEU, Denmark

(Best: this is different from the EU, UK test case)Reason: because conversion rates for Pound and Kroner are different

EU currency compliant countriesNon-EU countries

Best: one for each country; see aboveIllegal or unrecognized country

When to Use

When there are natural groupsAND your are confident their behavior is the same

Limited use for numerical rangesA more thorough technique coming up next



48

Strengths and Weaknesses

+ Powerful test case reduction technique+ Generally easy to do- Can miss weird exceptions in the code

Unit 4.2.2

Testing Smart:Boundary Value Testing



49

General Concept

Errors tend to cluster at the edgesFor example, “off by one” errorsSo we test thereJust below, just above

Appropriate for numeric valuesImplicitly includes equivalence classes

For those values

Steps

1. Identify the numeric equivalence class2. For each (numeric) boundary:

1. Write a test immediately below it2. Write a test immediately above it3. If appropriate, exactly on the boundary



50

What does “Immediately”mean?

One Unit AwayFor whatever unit it usesIntegers: straightforwardFloating point: hard

May be machine/compiler dependentMost likely: you have to decide on a reasonable accuracy threshold – just like you (should) always do

Exercise 4.3:

Define “one unit away” for the following:

US currencyAge (for ticket discounts)Expiration dateTime



51

Exercise 4.3 Answers

Define “one unit away” for the following:US currency

One cent

Age (for ticket discounts)One year

Expiration dateOne day

TimeDepends on what it is used for!

Exercise 4.4: Ticket Prices

Under 5: freeChild ages 5 to 12: $5.00Student ages 13 to 18: $7.50Adult ages 18 to 64: $10.00Senior ages 65 to 80: $7.00

Note: be careful …



52

Exercise 4.4 AnswersTicket Prices

Cases:-1 (error)0 (free)4 (free)5 ($5.00)12 ($5.00)13 ($7.50)18 (ambiguous specification!)64 ($10.00)65 ($7.00)80 ($7.00)81 (missing specification)

Note: tests can uncover specification errors

Exercise 4.5: Big Sale!

Just one requirement: If you buy ten dollars or more, you get 10% off your bill

Easy, right?



53

Exercise 4.5: AnswersTrickier than meets the eye:Cases:

- $0.01 (error)$0.00 (no sale)$0.01 ($0.01)$9.99 ($9.99)$10.00 ($9.00)$10.04 ($9.00)$10.05 ($9.01 – ambiguous spec.)Is there a maximum? Yes! Limited by the size of integers on the machine.

Advantages

Focuses on error-prone areasProven to be valuable

Can uncover common specification problemsNo real disadvantages



54

Unit 5

Putting the Test into TDD

Keep TDD’s Strengths

Write tests firstAutomated test harness

Can be very simple

Execute tests frequently



55

Improve TDD’s Weaknesses

Systematic, comprehensive coverageConsider error cases systematicallyGood design up front

Not necessarily ‘Big’Reduce need for refactoring

Test-based TDD Steps

Design all your test cases firstFollow Use CasesEquivalence classes, Boundary valuesUse the patterns as guides

Use test cases to guide lightweight design“Things I have to think about”

Implement one test and code as beforeMay consider implementing more than one at a time – depends on the tests



56

Unit 6

Summary

One-Slide Review ☺Software testing is very difficultTesting by developers AND system testers necessaryTrain you mind to think like a tester

Follow the patternsWill also make you a better designer

Write tests systematicallyCyclomatic complexity as a guide to number of testsEquivalence classesBoundary values

Use systematic testing approach to TDD



57

Final Exam!

Give one (good) goal of testing:

Why is it impossible to completely test a system?

Exam, Page 2

What is the Al Capone principle?

Explain one or more developer testing patterns you might use:



58

Exam, Page 3: Write tests for the following system fragment:

The “withdraw cash” function of an ATM (the software part only):

User specifies from savings or checkingAmount is up to $200, in multiples of $20.Your software communicates with the bank, and then instructs the dispenser to dispense the proper amount of cashUser may ask for a receipt or not

Answers to Page 3:Equivalence classes:

Savings/CheckingReceipt yes/noMultiple of 20: yes/no

Boundary values:< $0 (may not be physically possible)$0$200> $200: $201 (Note: may consider $200.01 instead)> $200: $220== Balance in account== Balance in account - $ .01== Balance in account + $ .01



59

More Answers:

More test cases, based on the patterns:Incomplete transactions

User walks away (Creative Idiot User)Link goes down

Multiple accesses to same account simultaneouslyMultiple withdraws (did account get updated properly?)

Tutorial Completed!

Testing is very creative, challenging and fun!

Happy testing!



60

BibliographyMyers, Glenford L., et al, The Art of Sfotware Testing, 2nd rev. ed., Wiley, 2004.Kaner, Cem et al., Testing Computer Software, 2nd

ed., Wiley, 1999.Kener, Cem et al., Lessons Learned in Software Testing, Wiley, 2001.,McConnell, Steve, Code Complete, 2nd ed., Microsoft Press, 2004.Cockburn, Alistair, Agile Software Development: The Cooperative Game, 2nd ed., Addison-Wesley, 2007.



testing: it’s part of your job — make it effective!egu/eto2007/testy.pdf · challenges of...

Documents