reducing test case bloat2 1

White Master

Replace witha graphic

5.5” Tall & 4.3” Wide

Copyright 2010 Adobe Systems Incorporated. All rights reserved.

ReducingTest Case Bloat

Lanette CreamerAdobe Systems, Inc

QASIG March 2010

®

Copyright 2010 Adobe Systems Incorporated. All rights reserved..

Introduction

Tester and Test Lead at Adobe since 2000

September 2009-First Agile project

Python scripts, collaboration, user focus, testing the whole system, cat photos.

Dysfunctional fiefdoms, clip art, spiders, waiting, retesting.

®


Agenda

What is a Test Case? What is Test Case Bloat?

Methods to reduce test case bloat in practice.

Barriers to reducing bloat.

How did it go?

Q&A

®


Section One: Why reduce at all?

“I have no help to send, therefore I must go myself.”

J.R.R. Tolkien

®


We like our bloat

How is that working for you?

®


What is a Test?

A test is an action which produces discoveries that can be used to evaluate product quality.

®


Test Case

Is every test a test case?

®


What is Bloat?

Test Case Bloat-More test cases than you can effectively manage* in the time you have. Is all bloat test cases?No.

*Manage may imply some or all of the following: cover, update, execute, report on, allocate, run, automate, respond to.

®


Section Two: Find the bloat

“All we have to decide is what to do with the time that is given to us.”

J.R.R. TolkienThe Fellowship of the Ring 1954

®


Is it worth keeping?

Cost versus RiskIf it doesn’t result in a bug is it bloat?If it does result in a bug is it always the most important bug?

Does reduce mean delete?Is it worth keeping all test cases just in case?

Time

Culture

Set up

Trust

®


Section 3: Ideas used in practice with success

“Their plans were improved with the best advice.”

J.R.R. Tolkien

®


Ways to Reduce Test Case Bloat

Mathematical Combinations

• Scatterplot

• Pairwise

• Random Selection

• Rotation of Environmental Factors

®



Tiered by Importance

• Priority 1, 2, 3

• High, Medium, Low

• Daily, Build, Weekly, Once

Stakeholders Priorities

• Poker Chips

• Backlog

®


Ways to Reduce Test Case

Change Based

• Bug fixes

• Features added

• Components integrated

• New visualization tools

Timing Based

• Covered later elsewhere

• Stop running at ______

®



Test Case Find Results

• Areas of breakage

• Date of last failure

• Auto retire?

Model Based

• Interactions with more risk

• Changes to the model

®



Code Coverage Based

• Reconcile duplication

• Which code is covered

®



Customer Workflow Based

• Usage data

• Sales info

• Meeting users

• Studies

• Surveys

®


Section 4: Letting Go

“The world changes, and all that once was strong now proves unsure.”

J.R.R. Tolkien

®


Does it still matter?

Risk:

High defect density.

Developers are worried about it.

Area of integration.

Area undergoing change.

Provides code coverage.

Data showing it is used.

Bottom line impact.

®


Does it still matter?

Protecting Your Peeps:

Impacts a customer workflow.

Barrier to adoption.

Top customer issue.

Silent failure is catastrophic.

Stakeholders still care about it.

®


Letting Go

When I let go of what I am, I

become what I might be.

- Lao Tzu

You do not need a parachute

to skydive. You only need a

parachute to skydive twice.

-Unknown (likely a tester)

®


Section 5: Human Elements and Change Management

“Valour needs first strength, then a weapon.”

J.R.R. Tolkien

®


Why it may be hard

What if we choose wrong?

Management would rather hear that we ran out of time for the low priority tests rather than that we decided not to test them.

Not everyone agrees on what is important.

Some are ethically opposed to this idea.

No one wants to lose value.

®


Why it may be hard

Feels safer to just keep them.

We paid good money for these test cases!

We’ve invested too much.

No time to deal with this!

It would be wasteful.

We may need these some day.

®


Other Ideas

More AutomationOk, but is it maintainable?Will it have value in time?You are making it instead of?Who runs, isolates, owns, maintains it?What about changes to what it relies on?

More TestersOk, but are they trained?Do they save or cost time?Do they come with more tests implied?More or less risk?

®


Other Ideas

Do nothing• Status Quo may be ok.• Powerless to change process.• Quality not culturally important.• Bloat is acceptable.• Change not worth the political

cost.• Acceptance may be enough.

Bloat is the norm for the majority of people and now over 50% of all household pets. It is hard to change. Long term success is the most difficult part.

®


Section 6: “Go forth and be spectacular!”-Gretchen Sidener

“Still round the corner there may wait, A new road or a secret gate.”

J.R.R. Tolkien

®


Now What? Now success!

The hard partThere is no after. Only during.Evaluate what worked.Identify what didn’t apply.Test cases usually shouldn’t live forever.Pruning test cases is part of test planning.Know when to start over.Share your own ideas with others!

The good newsThere is no failure, only “another fine learning opportunity”.

®


Section 7: Disclaimer and Cautionary Note

“...for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill.”

J.R.R. Tolkien

®


Choose Wisely

The same level of risk is not appropriate for all situations.

Failure to choose gives you fewer options. This experience report is intended to expand your options, not as a directive.

These ideas have been tested in one context and worked well for my team. Use at your own discretion. No warrantees implied.

®


Summary

Bloat happens to good people, projects, and pets.

With courage, information, and your own plan you can deal with test case bloat on your own terms.

A whole community of testers is here to help.

Each iteration is a new chance.

®


Followup-Technical Details

V A R I AT I O N S O N B L O AT

Jean Hartmann’s work (Microsoft) see http://www.pnsqc.org/conference09/Paper_33_slides.pdf

Reducing Test Case Bloat Technical Paper, Version 1.0 http://www.pnsqc.org/conference09/Paper_25_slides.pdf

Visual Representations of Data http://marlenacompton.com/?p=553

http://www.pnsqc.org/conference09/Paper_33_slides.pdf




http://marlenacompton.com/?p=553

White Master

Replace witha graphic

5.5” Tall & 4.3” Wide


Q&A

[email protected] @lanettecream Blog at http://blog.testyredhead.com

http://blog.testyredhead.com/

reducing test case bloat2 1

Technology

bloat test cases

test caseis

test lead

important bug

doesnt result

dysfunctional fiefdoms

clip art

user focus