Realism in Assessment of Effort Estimation Uncertainty:

It Matters How You AskBy Magne Jorgensen

IEEE Transactions on Software Engineering Vol. 30, No. 4, April 2004Presented by Debra Dirlam Oct 20 2004

Effort Estimation Uncertainty

• How sure are you of this estimate?

• Managers depend of your estimate and your level of uncertainty about the estimate.

– For sureness in manager’s decisions– For bidding on contracts– For project contingency buffers

It matters how you ask

• How should you frame your request for uncertainty information about an estimate?

• Obvious wrong way:

“You don’t believe that it will take you more than 1700 hours, do you?”

Traditional Framing of the question

• Estimators are asked to provide the minimum and maximum effort values based in given confidence levels

• Confidence level usually 90%

• “What is the minimum and maximum effort and be 90% sure?”

Alternative Framing of the question

• Estimators are asked to assess the probability of the actual effort being higher or lower than a certain value.

“How likely is it that the project will take more than 1700 hours?”

To Prove:

• Alternate framing provides greater realism and more useful information

• Traditional: Give me an estimate that you are 90% certain.

• Alternate: Give me an estimate and tell me your certainty.

Research Steps

Step 1• Identify the size of the systematic

overconfidence & understand the reasons– Found overconfidence high– When estimates claimed to be 90% confident

they actually were only 60% on target– Level of overconfidence supported by other

studies– Reasons… later

Research Step 2

• Looked at formal effort estimation uncertainty models designed to replace expert judgment– Some models could remove overconfidence

at expense of widening the min-max interval– Conclude that current models could not

replace expert judgment– More promising approach is to support expert

judgment

Research Step 3

• Evaluated several strategies for judgment support in student experiments

• One evaluated the framing variant and gave promising results

• The experiment was replicated with software professionals

The Software Professionals Experiment• 29 experienced software developers & project managers • Paid to participate• Divided randomly into 2 groups

• After giving estimate of most likely effort, half were asked

“Tell me the interval in which you are 90% confident”

And the other half were asked

“Tell me the probability that the actual effort will be between 50% to 200% of your estimate”

• 10 real world software projects were estimated

• Used expert judgment and an “experience database” of 5 similar projects

• Feedback given after each estimate

• Asked to reflect on performance

The Software Professionals Experiment - Training

The Software Professionals Experiment – The Estimations

• 30 software enhancement tasks previously conducted in a large telecom company

• Estimate of 1st task was based on an “experience database” of 5 previously completed tasks

• Estimate of 2nd task was based on the “experience database” and the feedback of the 1st task

Results

• A hit rate similar to average confidence indicates good correspondence

• Traditional framing shows slow approach to correspondence

• Alternative framing shows a close to perfect correspondence on all sequence of tasks.

4th Step – Full Scale Industrial Experiment

• 2 medium sized Norwegian software development companies

• No formal estimation process in place

• All estimates based on expert judgment

• Company projects and employees were similar

• 18 months, projects >10 hours < 8 months

• Projects were independent of each other

Industrial Experiment Design• During estimation phase, asked to

complete questionnaire on effort estimation uncertainty assessment with either Traditional or Alternative framing.

• Framing type was randomly chosen with 47 traditional framings and 23 alternative framings for a total of 70 projects

• Possible for an estimator to have chance to do both framings

• No feedback

Industrial Experiment Results

• Results were similar to previous experiment

• Traditional: 90% confidence corresponded to 74% hit rate

• Correspondence better in the Alternative framing: 87% hit rate to 88% confidence

• Analysis of any systematic favoritism: none or against Alternative framing

Discussion of Results

• The 2 framing provide the same statistical problem

• Looks how software professionals perceive and perform the uncertainty tasks in the 2 framings– Finds 2 important differences

Differences in How Software Professionals Perceive and Perform in the 2 Framings

• Seems to be a better fit between Alternative framing and the format of historical estimation data.

• The Traditional framing requires more complex analytical skill

• Uncertainty estimates are highly intuitive

Differences in How Software Professionals Perceive and Perform in the 2 Framings

• Software professionals may have goals other than realism in uncertainty estimates

• Worry about providing meaningless wide intervals

• Providing narrower intervals and more confidence evaluates to more skill as mistakenly perceived by managers

• In Alternative framing the interval is not provided by the estimator and cannot be used in skill evaluation

Limitations

• Cannot be generalize to all contexts

• Evaluated only high confidence uncertainty assessments for the Traditional framing and wide ranges for Alternative framing. Results may be different for other values.

• Not enough realism. Used questionnaires, importance of the role of requestor.

Conclusions

• The best approach to assessing the uncertainty of effort estimates depends on many factors– Skill of estimators– Availability of information about previous

projects– Type of information about the project to be

estimated– Other factors

Conclusion - continued

• The variety of factors does not lead to general laws to govern the assessment

• The lack of general laws does not mean that all choices are equally good

• The use of alternative framing is better supported by empirical evidence than the use of traditional framing.

• Use Alternative framing