traps in test estimation

8/7/2019 Traps in Test Estimation

http://slidepdf.com/reader/full/traps-in-test-estimation 1/18

Traps in Test EstimationTraps in Test Estimation

Shrini KulkarniShrini KulkarniPrincipal ConsultantPrincipal Consultant -- TestingTesting

iGATE Global SolutionsiGATE Global SolutionsBangalore, IndiaBangalore, India

[email protected]@igate.com

http://shrinik.blogspot.comhttp://shrinik.blogspot.com



What is a Trap and what is anEstimation?

Trap: noun - Something (often something deceptively attractive) that catches you unaware or place of confining or embarrassing

situation

Estimation: An approximate calculation of quantity or degree orworth of some activity or an object

of value

Test Estimation: An Act or processof estimation applied to a testing task



Consider this « A typical situation

Test Size

Test Effort

Schedule

Key QuestionKey Question ±± How many testers for how long? How many testers for how long?

Inputs

R equirements Test cases Use cases

User guide Expert Opinions



I s there any Scientific Method for Test estimation? I s there any Scientific Method for Test estimation?

A List of Commonly Used Techniques «A List of Commonly Used Techniques «

Guestimation ± Wild Ass Guess ± rule of thumb

A percentage of development effort

R isk Based methods ± Determine what to test and how much

Use of historical Data + Industry Standards

Work-break down Structure ± WBS method

Use of Dev Size estimates ± Use case, function points, COCOMO

Test point Analysis method

Delphi ± Wideband Delphi Method with WBSConsensus opinion in a team of expert estimators



But what all is included in ³Product´ andBut what all is included in ³Product´ andin the phrase ³test´ ?in the phrase ³test´ ?

How long it takes to youHow long it takes to youtest this Product ?test this Product ?

It all starts with this Motherhood Question «It all starts with this Motherhood Question «



Getting Trapped?Getting Trapped?

How will you know you will be able to complete Testing as per the

estimate ? We d on't. Why pret end we d o? We d on't. Why pret end we d o? - Matthew Heusser



Trap # 1Trap # 1 -- Model of TestingModel of Testing

There's a slippery slope between asking for good faith estimates ("Knowing what

you know now, when do you think you can deliver?") and predicting the future.

- Mathew Heusser

A simplified model of Testing is assumed.

Testing is an open ended search for problems ± decision to

stop testing is often decided by the stakeholders depending upon the parameters that are out side the purview of testing

Actual Testing model is complex hence estimates tend to gowrong

Testing is assumed to be complete when planned test casesare executed

Dependencies on environment, availability of Stable build totest and test data are typically ignored or down played



Trap # 3Trap # 3 -- Unknown FutureUnknown Future

W e¶re too optimistic, with short memories that mask the painful overruns

from previous projects ± Karl E. W iegers

Things change during the course of the project in unimaginable ways

Each test cycle presents its own challenges and issues

There might be be too many or too less bugs discovered in any cycle

There might be developer delays that eats up test time

There is might be pressure to skip tests and constantly change test sets

Development might lag behind bug find rate

There might be requirements changes hence development delays

There might be f ailures in Build verification tests

There might be significant number of ³non reproducible defects´



Trap # 4Trap # 4 -- Too many VariablesToo many Variables

G iving an Estimate would mean a commitment ± which you are not

sure about fulfilling ± that is a trap

Test scope ± that keeps changing depending upon available timefor testing

Set of features available in a given build Time available for testing for any given cycle Test environment availability

Stability of build for testing Unknown number of bugs discovered any given cycle Variable bug find and fix rate Additional cycles of Testing for every new bug or regression bug



Trap # 5Trap # 5 ±± Match that MAGIC numberMatch that MAGIC number

Role Play «Role Play «

U nlike other forms of estimation, I can make the estimate totally

predictable: I'll stop when the allotted time is up ± Michael Bolton



Michael Bolton¶s Recipe for Estimation «Michael Bolton¶s Recipe for Estimation «

Testing is an open ended search for problems ± we would never say ³we are

Done´ ± instead we temporarily stop and pass on the information to the

Stakeholder and wait for next opportunity to start «.

When do we stop then ? Several ways «.

Please see Next slide «



Heuristics for ³Stopping Testing´Heuristics for ³Stopping Testing´

� If we've modeled the product in lots of different ways, and then tested in accordance with those models, we can say that we've done enough testing and stop.

� If we're satisfied that we have addressed the list of compelling questions that we set outto answer (along with the other questions that we realized are important along the way), then we can say that we've done enough testing and stop.

� If the product is so horribly broken that it has to be send in for major rework, then we can

say that we've done enough testing and stop.

� If management decides that it has sufficient confidence to ship, and ships, then we cansay that we've done enough testing and stop.

� If management decides that it must ship the product, even though confidence in its quality is less than what we'd like it to be, we can say that we've done enough testing and stop.

� If we're testing on behalf of some one who is trying to decide whether the software isacceptable, and they say it is, we can say that we've done enough testing and stop.

� If we're testing on behalf of someone who is trying to decide whether the software is acceptable, and they say it isn't, until we get another version, we can say that we've

done enough testing and stop.



So « what is the solution?

I hear you, I hear you«´ So Mr. Big Man, what¶ syour solution?´ «.. the truth is

I don¶t have one.

I don¶t think that¶s a failure, but a realization ±

I¶m not the lone ranger, and the estimation

problem may be a bugbear, but it¶ s not awerewolf .Source : htt p://bl og.geeksmi t hol ogy .c om/2006/11/14/est imat i ng-is-bunk

But I do have some suggestions «

Ask what is in your sponsor's mind ± what date?

Take overall testing as cycle by cycle

Propose an Adaptive and iterative approach R esist the temptation to commit for ONE date

Say ± you will test until ship date and until told not to test«

H ell, t here are no r ules here± we¶re t ryi ng to acc omplish somet hi ng. -

T homas A. Edison



And «And «



Questions?Questions?



Thank youThank you

http://www.igate.com

http://shrinik.blogspot.com

[email protected]

[email protected]

F or f ur t her disc ussi ons:

Shrini Kulkarni

Principal Consultant - Testing

iGATE Global Solutions Ltd

Bangalore



ReferencesReferences

R apid Test Estimation ± Michael Bolton¶ s blog post

http://www.developsense.com/2007/01/test-project-estimation-rapid-way.html

traps in test estimation

Documents