sim presentation 092402

Copyright, Robust Decisions, 2002

www.robustdecisions.com

1

Increasing Business Value with Decision Management

David G. UllmanPresident, Robust Decisions Inc.

Jeffrey Barnes President, Compelevent Solutions

Pr



2

Goals

• Define Decision Management

• Describe major Decision Management objectives• Managing uncertainty

• Basing decisions on satisfaction and risk

• Deciding what to do next

• Demonstrate achieving the objects using an RFP example

• Discuss how Decision Management can help you produce more successful decisions



3

The value of information

DECISIONDECISIONDECISIONDECISION

JudgmentJudgmentJudgmentJudgment

KNOWLEDGEKNOWLEDGEKNOWLEDGEKNOWLEDGE

BehaviorBehaviorBehaviorBehavior

MODELSMODELSMODELSMODELS

RelationshipsRelationshipsRelationshipsRelationships

DATADATADATADATA

Increasing value



4

What makes critical decisions so difficult?

• There are many stakeholders: customers, influencers, decision makers and approvers

• Their knowledge is uncertain: qualitative, evolving and incomplete

• Their perspectives about what is important varies

• They must base decisions on expected satisfaction and assessment of risk

• Not clear what to do next "The only real problem in life is what to do next."

Arthur C. Clark



5

A process that ensuresstakeholder perspectives andtheir uncertain knowledge

are effectively utilized in determining what to do next to make thebest possible choices with

known expected satisfaction and risk

Given situation

Goal

What is Decision Management?



6

• Decisions that were declared final are later revisited • Team members don’t buy into decisions• Team decisions are made by edict or by loudest voice• Not all the expertise of the team members is utilized. • There is little confidence in decisions based on

uncertain information and unknown risk• Decisions are not justified, documented, or reused

Symptoms of Poor Decision Management



7

Cost of poor decisions

Product development:

• A poor decision - 20 people spend one week reworking a critical problem: $75,000 wasted on burdened salaries alone

Product procurement:

• A poor decision - Wastes product acquisition/implementation costs and creates excessive support costs. Expected added business value delayed, diminished or not realized entirely.



8

Typical decision problems

• IT investment portfolio

• Outsourcing decisions

• Vendor selection

• Hiring decisions

• Product development

• Partner selection

• Business strategy

• Washing machine selection



9

Simple example: RFP evaluation

• RFP contains• Detailed list of requirements that define the

ideal system - developed by functional representatives (stakeholders) in the organization

• The responses• 4 proposals submitted

• Your organization has dealt with some of the vendors before



10

Evaluation criteria given in the RFP

• Cost Criteria, < $100K 30%

• Technical Criteria 50%• System response time, <3 sec

• Training time, > 95% proficiency in 72 hrs

• Ease of use, > 75% satisfaction

• Risk of not meeting criteria, low

• ………..

• Management Criteria 20%• Strong management team

• Experienced with similar problems

• ……….. Total 100%

Whose weightings?

Is risk a useful criterion?

How do you manage qualitative criteria?(they are often the

discriminators)

What if one vendor proposes 90%?



11

Basic evaluation methods

• Fiat

• Loudest voice

• Competition

• Voting

• Inertia

• Omphaloskepsis



12

A more sophisticated evaluation method(The Decision Matrix)

Cost

Performance Management

Response time

Training time

Ease of use

Strong mgt team

Proj. Team experience

Total sum/wt

wt .3 .5 .2 1.0

Vendor 1 4 3 2 1 3 3 16/2.8

Vendor 2 4 4 4 3 5 4 24/3.9

Vendor 3 4 3 4 4 4 4 23/3.8

Vendor 4 4 5 5 4 2 2 22/3.9

“The Mechanical Design Process” 3rd

edition, David G. Ullman, McGraw Hill, 2003

Which will you choose?

Measures of satisfaction

Confidence vendor meets criterion5 = very high1 = very low



13

Decision Management improving the decision matrix

• Minimal additional work

• Inclusion of stakeholders perspectives

• Management of uncertain knowledge and differing perspectives about what is important

• Real time analysis of satisfaction and risk

• Guidance about what to do next to find the vendor most likely to be the best choice

• Capture and document the deliberation for justification and reuse

First topic



14

Uncertain evaluation

CostResponse

timeTraining

timeEase of

useStrong

mgt teamProj. Team experience

Evaluator 1 4 3 2 4 4 3

Evaluator 2 4 4 3 2 3 3

Evaluator 3 4 3 4 2 1 3

Evaluator 4 4 4 5 1 4 2

How do you manage the differences?Confidence that

vendor meets criterion5 = very high1 = very low

For Vendor 1



15

How do you manage the differences?

Average?

Discussion?

Voting?

– Averages are always brown.

– Discussions can be dominated by loudest voices.

– How can you account for knowledge differences?

– Move to Florida!

What about uncertainty?



16

The effect of knowledge in qualitative evaluations

Strong Mgt Team

Evaluator 1 4

Evaluator 2 3

Evaluator 3 1

Evaluator 4 4

Team score including knowledge = 4.1

For Vendor 1 Knowledge scaleExpertExperiencedInformedAmateurWeakUnknowledgeable

Average = 3

Knowledge

Experienced

Informed

Weak

Experienced

Confidence that vendor meets criterion5 = very high1 = very low



17

The effects of uncertainty during quantitative evaluations

• RFP - Training time > 95% proficiency in 72 hours

• In reality you will be delighted with 95%, but not sure it is achievable. Disgusted if < 85%

• Vendor bids - 95%

• Vendor’s reality – sure she can get 91% + 5%

• Your estimate of vendor’s reality is 87% + 8%

• If you could run an evaluation experiment you might find 90% + 8%

• After the system is in use the reality is 94%

Target uncertainty

Evaluation uncertainty

Changes with time



18

Next topic





• Guidance about what to do next to find the vendor most likely to be the best choice





19

• Risk is not a criterion!!

• Risk is a measure of uncertainty.

• Risk is one measure of how well each alternative meets each criterion.

Risk Analysis

• There is risk that the system response time will be >3 sec

• There is risk that the training time will be < 95% proficiency in 72 hrs

• There is risk that the ease of use will be < 75% satisfaction

• There is risk that the management team is not strong

• There is risk that they are not experienced with similar problems



20

There are two kinds of risk• Satisfaction Risk – The possibility that an

alternative will not meet a criterion ideal target

• Risk in choosing an alternative that can achieve only 93% proficiency

• Not really a risk measure, but lack of satisfaction

• Downside risk – The possibility that the actual satisfaction will be worse than expected

• We think that it will be 93% but there is a

chance it will only be 90%

• Also upside risk



21

Weighted Satisfaction

Vendor 1 16/2.8

Vendor 2 24/3.9

Vendor 3 23/3.8

Vendor 4 22/3.9

V4V2

3.9

V3 3.8

It is as important to understand the uncertainty as it is to believe the satisfaction!

Which one will you choose?

Downside risk

Ideal satisfaction - 4.0?

Satisfaction risk

Variation captured by the evaluations of different

stakeholders



22

Final topic





• Guidance about what to do next to find the vendormost likely to be the best choice





23

Traditionally

• Choose vendor with highest score (satisfaction)

• Loose sleep

• Re-bid

"The only real problem in life is what to do next." Arthur C. Clark

Vendor 1 16/2.8

Vendor 2 24/3.9

Vendor 3 23/3.8

Vendor 4 22/3.9

If you choose not to decide, you still have made a choice. Neil Peart,

drummer and lyricist for Rush.



24

• Satisfaction

• Value of information analysis (a sensitivity analysis)

• Level of consensus

• Level of knowledge or uncertainty

Decision Management basis for “what to do next” analysis

Traditionally all that is available

The needed richness

supplied by decision

management tools



25

1. Choose an alternative (The team has reached consensus, from all perspectives on a specific alternative)

2. Work toward consensus (Increase team consensus on identified critical evaluations)

3. Increase knowledge (Decrease uncertainty on identified critical evaluations)

4. Refine criteria (Identified critical criteria need to be refined)

Decision Management suggests



26

A “What to do next report”

Build consensus on Strong management team for Vendor 1

Build consensus on System response time for Vendor 2

Increase team knowledge about Ease of use for Vendor 1

Increase team knowledge about Strong management teamfor Vendor 2

Refine team experience criterion



27

The deliberation

• Build consensus Strong management team for Vendor 1

• Build consensus System response time for Vendor 2

• Increase knowledgeabout Ease of use for Vendor 3

Based on results team recommends Vendor 3 to decision maker

New what to next report

Initial what to next report



28

A process that ensuresstakeholder perspectives andtheir uncertain knowledge

are effectively utilized in determining what to do next to make thebest possible choices with

known expected satisfaction and risk

What is Decision Management?



29

On the next project

You will• Have more successful decisions

• Get more buy-in from stakeholders

• Keep out of trouble

• Document how decisions were reached for justification and reuse

• Deliver on promises made in the business case for the RFP

• Manage uncertainties

• Integrate stakeholder knowledge and perspectives

• Calculate satisfaction and risk

• Analyze what to do next



30

Tools that support decision

management

• @RISK, Palisades, http://www.palisade.com

• Expert Choice, http://www.expertchoice.com

• Accord™ www.robustdecisions.com



31

You now get to make the decision…

“What to do next”



32

For RFP problem.



33

Robust Decisions Inc.Delivering decision management consulting, training and tools.

Developers of Accord™ the real time, easy to use decision management application for distributed or co-located teams

sim presentation 092402

Documents