Copyright © 1996-2002, Satisfice, Inc.

v1.4.3 I grant permission to make digital or hard copies of this work for personal or

classroom use, provided that (a) Copies are not made or distributed for profit or commercial advantage, (b) Copies bear this notice and full citation on the first page, and if you distribute the work in portions, the notice and citation must appear on the first page of each portion. Abstracting with credit is permitted. The proper citation for this work is Rapid Software Testing (course notes, Fall 2002), www.testing-education.org, (c) Each page that you use from this work must bear the notice "Copyright (c) James Bach, james@satisfice.com” or, if you modify the page, "Modified slide, originally from James Bach", and (d) If a substantial portion of a course that you teach is derived from these notes, advertisements of that course must include the statement, "Partially based on materials provided by James Bach." To copy otherwise, to republish or post on servers, or to distribute to lists requires prior specific permission and a fee. Request permission to republish from James Bach, james@satisfice.com.

Acknowledgements

Some of this material was developed in collaboration with Dr. Cem Kaner, of the Florida Institute of Technology.

Many of the ideas in this presentation were reviewed and extended by my colleagues at the 7th Los Altos Workshop on Software Testing. I appreciate the assistance of the LAWST 7 attendees: Cem Kaner, Brian Lawrence, III, Jack Falk, Drew Pritsker, Jim Bampos, Bob Johnson, Doug Hoffman, Chris Agruss, Dave Gelperin, Melora Svoboda, Jeff Payne, James Tierney, Hung Nguyen, Harry Robinson, Elisabeth Hendrickson, Noel Nyman, Bret Pettichord, & Rodney Wilson.

Assumptions

You test software. You have at least some control over the design of

your tests and some time to create new tests. One of your goals is to find important bugs fast. You test things under conditions of uncertainty

and time pressure. You have control over how you think and what

you think about. You want to get very good at software testing.

Goal of this Class

This class is about how to test a productwhen you have to test it right now,under conditions of uncertainty,

in a way that stands up to scrutiny.

(This skill comes in handy even whenyou have more time, more certainty,

and are subject only to your own scrutiny.)

Internet time!

Less time than you’d like. Faster is better. Your plans will be interrupted. You are not fully in control of your time.

What is Internet time like?

The Challenge

When the ball comes to you…Do you know you have the ball?

Can you receive the pass?

Do you know what yourrole and mission is?

Do you know whereyour teammates are?

Are you ready to act, right now?

Can you let your teammates help you?

Do you know your options?

Is your equipment ready?

Can you read the situation on the field?

Are you aware of the criticality of the situation?

Your Moves:Rapid Testing Cycle

Make sense ofyour status

Compare statusagainst mission

Focus on whatneeds doing

START STOPDo a burst oftesting

Report

What About “Slow” Testing?

All Testing

Rapid

automationextensive

preparationsuper

testabilitysuperskill

Rigorousor Thorough

Management likesto talk about this…but they don’tfund it.

You can do this,no matter what.

KEY IDEA

TechnicalKnowledge

Specifications

Product

Ah! Problem!

Problem

Testing is in Your Head

Coverage

ProblemReport

Communication

The important parts of testing don’ttake place in the computer or on your desk.

CriticalThinking

DomainKnowledge

Experience

Welcome to Epistemologythe study of knowledge

Epistemology

Epistemology is the study of how we knowwhat we know. The philosophy of science

belongs to Epistemology.

All good testers practice Epistemology.

Basic Skills of Epistemology

Ability to pose useful questions. Ability to observe what’s going on. Ability to describe what you perceive. Ability to think critically about what you know. Ability to recognize and manage bias. Ability to form and test conjectures. Ability to keep thinking despite already knowing. Ability to analyze someone else’s thinking.

Epistemology Lesson:Abductive Inference

1. Collect data.2. Find several explanations that account for the data.3. Find more data that is either consistent or

inconsistent with explanations.4. Choose the best explanation that accounts for the

important data, or keep searching.

Abductive inference means finding the best explanation for a set of data.

Epistemology Lesson:Conjecture and Refutation

We never know product quality for certain. We conjecture about quality. To conjecture is to

explore plausible realities. A good conjecture is falsifiable: we can imagine facts

that would refute the conjecture. A conjecture can never be proven, only corroborated.

Corroborating evidence is most interesting when gained as the result of a genuine attempt to refute.

Good testing is a serious attempt to refute and corroborate conjectures about product quality.

Epistemology of Magic

Our thinking is limited We misunderstand probabilities We use the wrong heuristics We lack specialized knowledge We forget details We don’t pay attention to the right things

The world is hidden states sequences processes attributes variables identities

Magic tricks workfor the same reasons

that bugs exist

Studying magic canhelp you developthe imagination

to find better bugs.

Sufficiently advancedtechnology is

indistinguishablefrom magic.

Tester vs. Tourist:Testers Ask Critical Questions

How well does the product work? How do you know how well it works? What evidence do you have about how it works? Is that evidence reliable and up to date? What does it mean for the product to “work”? What facts would cause you to believe that it doesn’t work? In what ways could it not work, yet seem to you that it does? In what ways could it work, yet seem to you that it doesn’t? What might cause the product not to work well (or at all)? What would cause you to suspect that it will soon stop working? Do other people use the product? How does it work for them? How important is it for this product to work? Are you qualified to answer these questions? Is anyone else qualified?

Heuristics:Generating Solutions Quickly

adjective:

“serving to discover.”

noun:

“a useful method that doesn’t always work.”

“Heuristic reasoning is not regarded as final and strictbut as provisional and plausible only, whose purpose is to discover the solution to the present problem.”

- George Polya, How to Solve It

Some Everyday Heuristics

It’s dangerous to drink and drive. A bird in hand is worth two in the bush. Nothing ventured, nothing gained. Sometimes people stash their passwords near their

computers. Try looking there. Stores are open later during the Holidays. If your computer is behaving strangely, try

rebooting. If it’s very strange, reinstall Windows. If it’s a genuinely important task, your boss will

follow-up, otherwise, you can ignore it.

How Heuristics Differ fromOther Procedures or Methods

Heuristics are known to be wrong, at least some of the time.

No one can say for sure when a heuristic will work.

Heuristics aid or focus an open-ended problem-solving or solution-searching effort.

Heuristics can substitute for complete or rigorous analysis.

Tunnel-Vision is OurGreat Occupational Hazard

Problems you

can find with your

biases…

invisible

problems

invisible

problems

A Tester’s Attitude

Cautious Jump to conjectures, not conclusions. Practice admitting “I don’t know.” Have someone check your work.

Curious What would happen if…? How does that work? Why did that happen?

Critical Proceed by conjecture and refutation. Actively seek counter-evidence.

Good testersare hard to fool.

(and Courageous)

Important Vocabulary

Abductive inference is a way to make respected guesses. Conjectures (a.k.a. hypotheses) help us remember that we

may be wrong about our beliefs. Refutation means checking how you could be wrong,

before trusting a conjecture. Evidence is either consistent with (corroborates) or

inconsistent with a given conjecture. Heuristics help us get the right ideas at the right times. Raising questions and issues is the heart of testing.

The Triangle Program

This program takes three numbers as input. The numbers represent the dimensions of a

triangle. When you click on the check button, the program

tells you what kind of triangle the sides represent: scalene (no side equal to any other) isosceles (two sides are equal) equilateral (all sides are equal)

I want you to test this program.Any questions?

KEY IDEA

The Universal Test Procedure

“Try it and see if it works.”

Learn about it Model it Speculate about it Configure it Operate it

Know whatto look for

See what’sthere

Understand the requirements

Identify problems Distinguish bad

problems from not-so-bad problems

Models EvaluationCoverage

All Product Testing is Something Like This

ProjectEnvironment

ProductElements

QualityCriteria

TestTechniques

PerceivedQuality

The Five Problems of Testing

ProjectEnvironment

ProductElements

QualityCriteria

TestTechniques

PerceivedQuality

Logistics Problem

Coverage ProblemEvaluation

Problem

ReportingProblem

Stopping Problem

Models

A Model is… A map of a territory A simplified perspective A relationship of ideas An incomplete representation of reality A diagram, list, outline, matrix…

No good test design has ever been done without models.

The trick is to become aware of how you modelthe product, and learn different ways of modeling.

Coverage

There are as many kinds of coverage as there are ways to model the product. Structural Functional Data Platform Operations

Product coverage is the proportion of theproduct that has been tested.

An evaluation strategy ishow you know the product works.

“...it works.”

really means

“...it appeared to meet some requirement to some degree.”

Which requirement? Appeared how?To what degree? Perfectly? Just barely?

Rapid Evaluation:“HICCUPP”

Consistent with History: Present function behavior is consistent with past behavior.

Consistent with our Image: Function behavior is consistent with an image that the organization wants to project.

Consistent with Comparable Products: Function behavior is consistent with that of similar functions in comparable products.

Consistent with Claims: Function behavior is consistent with what people say it’s supposed to be.

Consistent with User’s Expectations: Function behavior is consistent with what we think users want.

Consistent within Product: Function behavior is consistent with behavior of comparable functions or functional patterns within the product.

Consistent with Purpose: Function behavior is consistent with apparent purpose.

Exercise

Explain what you covered and what your evaluation strategies were for triangle.

KEY IDEA

Test Cycle:What You Do With a Build

1.Receive the product. Formal builds Informal builds Save old builds.

2. Clean your system. Completely uninstall earlier builds.

3.Verify testability. Smoke testing Suspend test cycle if the product is untestable.

Test Cycle:What You Do With a Build

4.Determine what is new or changed. Change log

5.Determine what has been fixed. Bug tracking system

6.Test fixes. Many fixes fail! Also test nearby functionality.

7.Test new or changed areas. Exploratory testing.

Test Cycle:What You Do With a Build

8.Perform regression testing. Not performed for an incremental cycle. Automated vs. manual Important tests first!

9.Report results. Coverage Observations Bug status (new, existing, reopened, closed) Assessment of quality Assessment of testability

The Test Cycle

Time

Qu

alit

y

Typical Quality Growth

Ship it!

Risk Focus:Common and Critical Cases

Core functions: the critical and the popular. Capabilities: can the functions work at all? Common situations: popular data and pathways. Common threats: likely stress and error situations. User impact: failures that would do a lot of damage. Most wanted: problems of special interest to

someone else on the team.

Test Cycle Convergence

Time

Qu

ality

Good Enough Quality

Build

Cycle EndCycle Start

Tests failed: known bad

Tests passed: known good

Tests not run:Unknown}

Ideal Test Cycle Convergence

Untestable Quality

Rapid Bug Investigation

Identification Notice a problem. Recall what you were doing just prior to the problem. Examine symptoms of the problem w/o disturbing system state. Consider possibility of tester error.

Investigation How can the problem be reproduced? What are the symptoms of the problem? How severe could the problem be? What might be causing the problem?

Reality Check Do we know enough about the problem to report it? Is it important to investigate this problem right now? Is this problem, or any variant of it, already known? How do we know this is really a problem? Is there someone else who can help us?

Identify

Investigate Check

Exercise

“I’ve modified the Triangle program so that it now renders an image of the triangle. I’m also worried about the limitations of the input field. Please do another test cycle on it.”

-- Your Friendly Programmer

Exercise

Provide a clear report on how completelyyou tested Triangle

An Innocent Question from Management…

“On a scale of 0 to 100, how completely did you test the Triangle program?”

completelyuntested

completelytested

0 10050

1. Answer with a number.

2. You can also offer a comment, if you have one.

3. Any questions?

What Could We Report?

Tasks: Things we are doing. Example: “We are trying to reinstall the server so we can do the smoke tests. We have 28 fix verifications to get through.”

Product Coverage: What we have examined. Example: “We have tested printing, performance, and compatibility.”

Product Risk: Problems or potential for problems in the product. Example: “We have found 53 bugs.”

Agreement: What we specifically contracted to do. Example: “We have implemented 90% of the tests on our test plan. As we agreed, we have started testing for printer compatibility.”

What Could We Report?

Project: Schedule, documents, resources, people, or anything else that makes it possible for us to test. Example: “Testing is on schedule. But Jaeline goes on vacation next week.”

Mission: The ultimate goal, which may be to ship a product that meets quality criteria, satisfy customers, or some other goal that can be a guiding principle for the testing. Example: “We don’t yet know enough about feature X of the product.”

Client Satisfaction: What our clients think of our work. Example: “The project manager is happy with our work.”

Reporting Considerations

Reporter safety: What will they think if I made no progress? Client: Who am I reporting to and how do I relate to them? Rules: What rules and traditions are there for reporting, here? Significance of report: How will my report influence events? Subject of report: On what am I reporting? Other agents reporting: How do other reports affect mine? Medium: How will my report be seen, heard, and touched? Precision and confidence levels: What distinctions make a

difference?

Take responsibility for the communication.

KEY IDEA

Test Strategy

“How we plan to cover the product so as to develop an adequate assessment of quality.”

A good test strategy is: Product-Specific Risk-focused Diversified Practical

Test Strategy

Test Approach and Test Architecture are other terms commonly used to describe what I’m calling test strategy.

Example of a poorly stated (and probably poorly conceived) test strategy: “We will use black box testing, cause-effect graphing,

boundary testing, and white box testing to test this product against its specification.”

Test Strategy

Makes use of test techniques. May be expressed by test procedures and cases. Not to be confused with test logistics, which

involve the details of bringing resources to bear on the test strategy at the right time and place.

You don’t have to know the entire strategy in advance. The strategy should change as you learn more about the product and its problems.

Test Cases/Procedures

Test cases and procedures should manifest the test strategy.

If your strategy is to “execute the test suite I got from Joe Third-Party”, how does that answer the prime strategic questions: How will you cover the product and assess quality? How is that practical and justified with respect to the specifics of this

project and product?

If you don’t know, then your real strategy is that you’re trusting things to work out.

Exercise

Produce a test strategy for DecideRight

Test Strategy Heuristic:Diverse Half-Measures

There is no single technique that finds all bugs. We can’t do any technique perfectly. We can’t do all conceivable techniques.

Use “diverse half-measures”-- lots of differentpoints of view, approaches, techniques, evenif no one strategy is performed completely.

Strategy Heuristic:Function/Data Square

Data

Functions

risk testing

Function testing

reliabilitytesting

smoketesting

The Fallacy of Repeated Tests:Clearing Mines

mines

Totally Repeatable TestsWon’t Clear the Minefield

mines fixes

Variable Tests areTherefore More Effective

mines fixes

Exercise

Find at least two situations where the heuristic:

“Variable tests are better than repeated tests”

is WRONG.

Test Techniques

A test techniqueis a recipe

for performingthese tasks that

will reveal somethingworth reporting

Analyze the situation. Model the test space. Select what to cover. Select evaluation methods. Configure the test system. Operate the test system. Observe the test system. Evaluate the test results.

General Test Techniques

Function testing Domain testing Stress testing Flow testing Scenario testing User testing Regression testing Risk testing Claims testing Random Testing

For any one of these techniques,there’s somebody, somewhere,

who believes that is theonly way to test.

Function Testing

“March through the functions”

- A function is something the product can do.

- Identify each function and sub-function.

- Determine how you would know if they worked.

- Test each function, one at a time.

Key Idea:

Summary:

Good for: assessing capability rather than reliability

Domain Testing

“Divide the data, and conquer”

- A domain is a set of test data.

- Identify each domain of input and output.

- Analyze limits and properties of each domain.

- Identify combinations of domains to test.

- Select coverage strategy:

e.g., exhaustive, boundaries, best representative

Key Idea:

Summary:

Good for: all purposes

Stress Testing

“Overwhelm the product”

- Select test items and functions to stress.

- Identify data and platform elements that relate to them.

- Select or generate challenging data and platform configurations to test with:

e.g., large or complex data structures,

high loads, long test runs, many test cases

Key Idea:

Summary:

Good for: performance, reliability, and efficiency assessment

Flow Testing

“Do one thing after another”

- Define test procedures or high level cases that

incorporate many events and states.

- The events may be in series, parallel, or some combination thereof.- Don’t reset the system between events.

Key Idea:

Summary:

Good for: finding problems fast(however, bug analysis is more difficult)

Scenario Testing

“Test to a compelling story”

- Design tests that involve meaningful and complex interactions with the product.- A good scenario test is a plausible story of how someone who matters might do something that matters with the product.- Incorporate multiple elements of the product, and data that is realistically complex.

Key Idea:

Summary:

Good for: Finding problems that seem important

User Testing

“Involve the users”

- Identify categories of users.- Understand favored and disfavored users.-Find these users and have them do testing or help you design tests.-User testing is powerful when you involve a variety of users.

Key Idea:

Summary:

Good for: all purposes

Regression Testing

“Test the changes”

- Identify what product elements changed.

- Identify what elements could have been

impacted by the changes.

- Coverage strategy:

e.g., recent bug fixes, past bug fixes,

likely elements, all elements

Key Idea:

Summary:

Good for: managing risks related to product enhancement

Claims Testing

“Verify every claim”

- Identify specifications (implicit or explicit).

- Analyze individual claims about the product.

- Ask customer to clarify vague claims.

- Verify each claim.

- Expect the specification and product to be

brought into alignment.

Key Idea:

Summary:

Good for: simultaneously testing the product and specification, while refining expectations

Risk Testing

“Imagine a problem, then look for it.”

- What kinds of problems could the product have?- How would you detect them if they were there?- Make a list of interesting problems and design tests specifically to reveal them.- It may help to consult experts, design documentation, past bug reports, or apply risk heuristics.

Key Idea:

Summary:

Good for: making best use of testing resources;leveraging experience

Random Testing

“Run a million different tests”

- Look for an opportunity to automatically generate thousands of slightly different tests.

- Create an automated, high speed evaluation strategy.

- Write a program to generate, execute, and evaluate all the tests.

Key Idea:

Summary:

Good for: Assessing reliability across input and time.

KEY IDEA

Dynamic Quality Paradigm

Perfect

Awful

unnecessary quality

unacceptable quality

Item A

Item B

It’s more important to work on Item B.

Further improvement would not be a good use of resources.

Further improvement is necessary.

Good enoughquality bar

floating line

A Heuristic for Good Enough

1. X has sufficient benefits.

2. X has no critical problems.

3. Benefits of X sufficiently outweigh problems.

4. In the present situation, and all things considered, improving X would be more harmful than helpful.

BenefitsProblems

All conditionsmust apply.

Good Enough...

…with what level of confidence? …to meet ethical obligations? …in what time frame? …compared to what? …for what purpose? …or else what? …for whom?

Perspectiveis

Everything

Test Project Dynamics:Context Model

Test TeamExpertiseLoadingCohesionMotivationLeadershipProject Integration

Test LabTest PlatformsTest ToolsTest LibraryProblem Tracking SystemOffice Facilities

Product MissionStakeholdersQuality CriteriaReference Material

RequirementsProductProject LifecycleProject ManagementConfiguration ManagementDefect PreventionDevelopment Team

Development

MissionFind Important ProblemsAssess QualityCertify to StandardFulfill Process MandatesSatisfy StakeholdersAssure Accountability

Advise about QAAdvise about TestingAdvise about QualityMaximize EfficiencyMinimize CostMinimize Time

TestProcess

StrategyLogisticsWork-products

Test Project Dynamics:Context Model

T es t T eam T est L ab

RequirementsDevelopment

M iss ion

T estP roc ess

Test Project Dynamics:Context Model

GIV ENS GIV ENS

GIV ENSGIV ENS

M iss ion

CHOICES

CHOI

VENSMotivation

Capability

Test Project Dynamics:Givens vs. Choices

Motivation: What testing does the situation require? Capability: Can we perform that testing in this situation?

Tailoring the Test Process

Are GIVENS good enough?

Do CHOICES about process exploit the GIVENS and address the MISSION well enough?

Is MISSION achieved well enough?

How do you know?

MISSION:The most important part

Find important problems Assess quality Certify to standard Fulfill process mandates Satisfy stakeholders Assure accountability

Advise about QA Advise about testing Advise about quality Maximize efficiency Minimize time Minimize cost

The quality of testing depends on which of thesepossible missions matter and how they relate.

Many debates about the goodness of testingare really debates over missions and givens.

Slow down project toimprove testability

Driving analogy When on the freeway, go 65mph. When on a suburban artery, go 40mph. When in a neighborhood, go 25mph. When in a parking lot, go 5mph. Park very slowly.

Change Control Funnel The closer you are to shipping, the more reluctantly you

should change code. To minimize retesting, carefully review proposed

changes. “You can change what you like, but change may

invalidate everything we thought we knewabout quality.”

More risks,lower speeds.

time

“Why didn’t you find that bug?”

Highway Patrol analogy: Do you realizehow hard it would be to patrol the highways sothat all speeders are always caught?

Risk-based improvement argument: Our goal is to find important problems fast. Was that bug important? Could we economically have anticipated it? If so, we’ll modify our approach.

Testability reframe: We didn’t find it because the developers didn’t make the bug obvious enough for us to notice. The developers should put better bugs into the code. Let’s make a more testable product.

Exercise

We may want to purchase DiskMapper.

Analyze it and tell us what the testing issues might be.

KEY IDEA

Contrasting Approaches

In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they often are not recorded.

Product

Tests

Tests

(yes, this is a simplified view)

Forward-Backward Thinking

search for evidenceto corroborate or

refute the conjecture

discover evidenceand make conjectures

Evidence Conjectures

Lateral Thinking

but periodically take stockof your status against your mission

Let yourself be distracted…

‘cause you never knowwhat you’ll find…

Exploratory Testing Tasks

ExploreExecute

TestsDesignTests

Product(coverage)

Techniques

Quality(evaluation)

Discover the elements

of the product.

Discover howthe product

should work.

Discover testdesign techniquesthat can be used.

Decide whichelements to test.

Observe product behavior.

Speculate aboutpossible quality

problems.

Evaluatebehavior against

expectations.

Configure & operate the

product.

Select & applytest design techniques.

Testingnotes

TestsProblems

Found

Taking Notes

Test Coverage Outline/Matrix Evaluation Notes Risk/Strategy List Test Execution Log Issues, Questions & Anomalies

It would be easier to test if you changed/added… How does … work? Is this important to test? How should I test it? I saw something strange…

The Plunge in and QuitHeuristic

This is a tool for overcoming fear of complexity. Often, a testing problem isn’t as hard as it looks. Sometimes it is as hard as it looks, and you need to quit

for a while and consider how tackle the problem. It may take several plunge & quit cycles to do the job.

Whenever you are called upon to testsomething very complex or frightening, plunge in!

After a little while, if you are very confusedor find yourself stuck, quit!

Exploration Trigger Heuristic:No Questions

If you don’t have any issues or concernsabout product testability or test environment,

that itself may be a critical issue.

When you are uncautious, uncritical, or uncurious,that’s when you are most likely to let important

problems slip by. Don’t fall asleep!

If you find yourself without any questions,ask yourself “Why don’t I have any questions?”

Key Ideas ofExploratory Testing

The results from the tests you design and execute influence the next test you will choose to design and execute.

You build a mental model of the product while you test it. This model includes what the product is and how it behaves, and how it’s supposed to behave.

You test what you know about, and you are alert for clues about behaviors and aspects of the product that you don’t yet know about.

Doing Exploratory Testing

Keep your mission clearly in mind. Keep notes that help you report what you did,

why you did it, and support your assessment of product quality.

Keep track of questions and issues raised in your exploration.

To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.

Exercise

Write an explicit test procedure.

Make it at least 10 steps long.

Include coverage and evaluation instructions.

KEY IDEA

It Boils Down To…

YOU: Skills, equipment, experience, attitude

THE BALL: The product, testing tasks, bugs

YOUR TEAM: Coordination, roles, support

THE GAME: Risks, rewards, project environment, corporate environment, your mission as a tester

YOUR MOVES: How you spend your attention and energy to help your team win the game.

Rapid Testing

Develop your scientific mind. Know your coverage and evaluation strategy. Run crisp test cycles that focus first on areas of risk. Use a diversified test strategy that serves the mission. Assure that your testing fits the logistics of the project. Let your tests evolve as new information comes to light.

Copyright © 1996-2001, Satisfice, Inc.

v1.0

Rapid Test Management

Establish a strong and supportive testing role. Re-visit your strategy and mission every day. Advocate for bugs and testability. Maintain three lists: risks, issues, coverage. Test with your team. Continuously report the status and dynamics of

testing.

KEY IDEA

This is our role.We make informed decisions about quality possible.

We can do this under a variety of conditions.

Testers light the way.

This is the formula…

You must demand… complete specs and quantifiable criteria. protected schedule and early involvement. zero defect philosophy and control over release. resources to achieve complete test coverage.

…for hypochondria.

Look, testing is a difficult job.

“The perfect cure for hypochondria, 100 percent effective, is to contract a potentially fatal disease. It cures you instantly. It happened to me.”

Gene WeingartenThe Hypochondriac's Guide to Life and Death

That’s why they hire smart people, like us, to do it for them.

Too many textbooks seem to thinktesters are wind-up toys.

“This model identifies a standards-based life cycle testing process that concentrates on developing formal test documentation to implement repeatable structured testing on a software or hardware/software system. The general intent is that the test documentation be developed based on a formal requirements specification document...Once the documentation is developed, the test is executed.”

-- from a real article about testing. (I added the boldfacing to emphasize instructions)

Where’s the tester in this picture?

Testing skills compensate for a difficult project environment.

complete specs quantifiable criteria protected schedule early involvement zero defect philosophy control over release complete test coverage

– implicit specs & inference– meaningful criteria– risk-driven schedule– good working relationship– good enough quality– don’t be the gatekeeper– enough information

Instead of this… consider this.

Goal setting Risk identification

Task planning Risk planning

Task completion Risk reduction

failure

failure

success

success

We think about failure,and that can be “negative”.

Eight Commitmentsworth making to developers

We’ll test your code as soon as we can after it’s built. We’ll test important things first, and focus on important problems. We’ll write clear, thoughtful, and respectful problem reports. We’ll try not to be a bottleneck for development. We’ll tell you how we’re testing, and consider your suggestions. We’ll look for ways to test better and faster. We’ll try to accommodate how you like to work. We will not waste your time.

Our job is information.If they don’t want information…

Stop fretting and just be as helpful as you can. Let events play out. Carefully record what happens. Detach yourself from the outcome and refocus

on improving the next project, instead. Help the team use the experience as a

reference point for improvement.

use the Reality Steamroller technique:

KEY IDEA

Public Relations Problem

“What’s the status of testing?”

“What are you doing today?”

“When will you be finished?”

“Why is it taking so long?”

“Have you tested _______, yet?”

The Problem

Management has little patience for detailed test status reports.

Management doesn’t understand testing. Testing is confused with improving. Testing is considered a linear, independent task. Testing is assumed to be exhaustive. Testing is assumed to be continuous. Test results are assumed to stay valid over time. Impact of regression testing is not appreciated. Test metrics are hard to interpret.

A Solution

Report test cycle progress in a simple, structured way...

...that shows progress toward a goal... ...manages expectations... ...and inspires support... ...for an effective test process.

No process?

No Problem!Easy setup

The Dashboard Concept

Project conference roomLarge dedicated whiteboard

“Do Not Erase”

Project status meeting

Test Cycle Report

Test Effort

Test Coverage

Quality Assessment

Time

Product Areas

vs.

vs.

Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI

Efforthighlowlowlowblockedlowblockednonenonestart 3/17start 3/17low

C.11+22+12011003

Q. Comments

1345, 1363, 1401

automation broken

crashes: 1406, 1407

animation memory leak

new files not delivered

need help to test...

need help to test...

lab time is scheduled

Testing Dashboard Updated: Build:

2/21 38

Product Area

15-30 areas (keep it simple) Avoid sub-areas: they’re confusing. Areas should have roughly equal value. Areas together should be inclusive of

everything reasonably testable. “Product areas” can include tasks or risks-

but put them at the end. Minimize overlap between areas. Areas must "make sense" to your clients,

or they won’t use the board.

Areafile/editviewinsertformattoolsslideshowonline helpclipartconvertersinstallcompatibilitygeneral GUI

Test Effort

None

Start

Low

High

Pause

Blocked

Ship

Not testing; not planning to test.

Regression or spot testing only; maintaining coverage.

Focused testing effort; increasing coverage.

No testing yet, but expect to start soon.

Temporarily ceased testing, though area is testable.

Can’t effectively test, due to blocking problem.

Going through final tests and signoff procedure.

Test Effort

Use red to denote significant problems or stoppages, as in blocked, none, or pause.

Color ship green once the final tests are complete and everything else on that row is green.

Use neutral color (such as black or blue, but pick only one) for others, as in start, low, or high.

Test Coverage

0

1

1+

2

2+

3

We don’t have good information about this area.

More than sanity, but many functions not tested.

Common & Critical:

Sanity Check:

Some data, state, or error coverage beyond level 2.

Complex Cases:

all functions touched; common & critical tests executed.

strong data, state, error, or stress testing.

major functions & simple data.

Test Coverage

Color green if coverage level is acceptable for ship, otherwise color black.

Level 1 and 2 focus on functional requirements and capabilities: can this product work at all?

Level 2 may span 50%-90% code coverage. Level 2+ and 3 focus on information to judge

performance, reliability, compatibility, and other “ilities”: will this product work under realistic usage?

Level 3 or 3+ implies “if there were a bad bug in this area, we would probably know about it.”

Quality Assessment

“We know of no problems in this area that threaten to stop ship or interrupt testing, nor do we have any definite suspicions about any.”

“We know of problems that are possible showstoppers, or we suspect that there are important problems not yet discovered.”

“We know of problems in this area that definitely stop ship or interrupt testing.”

Use the comment field to explainanything colored red, or any non-green

quality indicator.

Comments

Problem ID numbers. Reasons for pausing, or delayed start. Nature of blocking problems. Why area is unstaffed.

Using the Dashboard

Updates: 2-5/week, or at each build, or prior to each project meeting.

Progress: Set expectation about the duration of the “Testing Clock” and how new builds reset it.

Justification: Be ready to justify the contents of any cell in the dashboard. The authority of the board depends upon meaningful, actionable content.

Going High Tech: Sure, you can put this on the web, but will anyone actually look at it???

KEY IDEA

Exploratory testing relies on tester intuition. It is unscripted and improvisational.

How do I, as test manager, understand what’s happening, so I can direct the work and defend it to my clients?

SKILLthere’s no shortcut

No one can read your mind.

You must gain the skill to explain your testing…so that you can be accountable for it.

That requires a lot of practice.In our experience: several months of daily practice.

This is a black box…Just like your mind.

Introducing the Test Session

1) Charter

2) Time Box

3) Reviewable Result

4) Debriefing vs.

Charter:A clear mission for the session

A charter may suggest what should be tested, how it should be tested, and what problems to look for.

A charter is not meant to be a detailed plan. General charters may be necessary at first:

“Analyze the Insert Picture function”

Specific charters provide better focus, but take more effort to design: “Test clip art insertion. Focus on stress and flow techniques, and

make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”

Time Box:Focused test effort of fixed duration

Brief enough for accurate reporting. Brief enough to allow flexible scheduling. Brief enough to allow course correction. Long enough to get solid testing done. Long enough for efficient debriefings. Beware of overly precise timing.

Short: 60 minutes (+-15)Normal: 90 minutes (+-

15)Long: 120 minutes (+-15)

Debriefing:Measurement begins with observation

The manager reviews session sheet to assure that he understands it and that it follows the protocol.

The tester answers any questions. Session metrics are checked. Charter may be adjusted. Session may be extended. New sessions may be chartered. Coaching happens.

Reviewable Result:A scannable session sheet

Charter #AREAS

Start Time Tester Name(s) Breakdown

#DURATION #TEST DESIGN AND EXECUTION #BUG INVESTIGATION AND REPORTING #SESSION SETUP #CHARTER/OPPORTUNITY

Data Files

Test Notes Bugs

#BUG

Issues #ISSUE

CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk. #AREASOS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional Analysis START-----------------------------------------------5/30/00 03:20 pm TESTER-----------------------------------------------Jonathan Bach TASK BREAKDOWN----------------------------------------------- #DURATIONshort #TEST DESIGN AND EXECUTION65 #BUG INVESTIGATION AND REPORTING25 #SESSION SETUP20 

The Breakdown MetricsTesting is like looking for worms

Test Design and Execution

Bug Investigation and Reporting

Session Setup

Reporting the TBS BreakdownA guess is okay, but follow the protocol

Test, Bug, and Setup are orthogonal categories. Estimate the percentage of charter work that fell into

each category. Nearest 5% or 10% is good enough. If activities are done simultaneously, report the

highest precedence activity. Precedence goes in order: T, B, then S. All we really want is to track interruptions to testing. Don’t include Opportunity Testing in the estimate.

Activity HierarchyAll test work fits here, somewhere

all work

non-session

session

opportunity on charter

test bug setup

inferred

Non-Session61%

Test28%

Bug4%

Opportunity1%

Setup6%

Work Breakdown:Diagnosing the productivity

Do these proportions make sense? How do they change over time? Is the reporting protocol being

followed?

0.0

50.0

100.0

150.0

200.0

250.0

300.0

5/26 6/9 6/23 7/7 7/21 8/4 8/18

Coverage:Specifying coverage areas

These are text labels listed in the Charter section of the session sheet. (e.g. “insert picture”)

Coverage areas can include anything areas of the product test configuration test strategies system configuration parameters

Use the debriefings to check the validity of the specified coverage areas.

Coverage:Are we testing the right stuff?

Is it a lop-sided set of coverage areas?

Is it distorted reporting?

Distribution of On Charter TestingAcross Areas

0

20

40

60

80

100

120

• Is this a risk-based test strategy?

Using the Data to Estimate a Test Cycle

1. How many perfect sessions (100% on-charter testing) does it take to do a cycle? (let’s say 40)

2. How many sessions can the team (of 4 testers) do per day? (let’s say 3 per day, per tester = 12)

3. How productive are the sessions? (let’s say 66% is on-charter test design and execution)

4. Estimate: 40 / (12 * .66) = 5 days5. We base the estimate on the data we’ve collected.

When any conditions or assumptions behind this estimate change, we will update the estimate.

Challenges ofHigh Accountability Exploratory Testing

Architecting the system of charters (test planning) Making time for debriefings Getting the metrics right Creating good test notes Keeping the technique from dominating the

testing Maintaining commitment to the approach

For example session sheets and metrics seehttp://www.satisfice.com/sbtm