tac how to measure anything

7/28/2019 TAC How to Measure Anything

1/23

Copyright 2007 Hubbard Decision Research and The Advisory Council, Inc.

How to Measure Anything

Doug HubbardExpert

The Advisory Councilwww.TACadvisory.com


2/23

Copyright 2007 Hubbard Decision Research and The Advisory Council, Inc. 2

How to Measure Anything I conducted 55 major risk/return analysis projects

so far that included a variety of impossiblemeasurements

I found such a high need formeasuring difficult things that Idecided I had to write a book

The book will be released inJuly 2007 with the publisher

John Wiley & Sons This is a sneak preview of

many of the methods in thebook


3/23


A Few Examples

Org. Problem Findings Benefit

EPA Assess value ofSafe Drinking

Water InformationSystem SDWIS

Identified andmeasured previously

unnoticed risks &benefits

$15 million improvedNPV from reprioritizing

functions

US MarineCorp

Forecast fuel usefor the battle field

Found factors thatbetter correlated toactual fuel use

$50 million savings peryear by reducingunnecessary battlefieldinventory

AmericanExpress

Analyze ROI of adedicated testnetwork for newapplications

Determined key metricto prioritizereplacements wasdowntime reduction

$2 million improved netbenefit reprioritizingrollout

Source: Hubbard Decision Research


4/23


A Few More Examples Risk of IT

The risk of obsolescence

The value of a humanlife

The value of saving anendangered species

The value of public

health The value of IQ points

lost by children exposedto methyl-mercury

The value of bettersecurity

The future demandfor space tourism

The value of betterinformation

The value ofinformationavailability

Productivity andperformance


5/23


Three Measurement Muses Eratosthenes measured the Earths

circumference to within 1% accuracy

Enrico Fermi the physicist who usedFermi Questions to break down anyuncertain quantity (and was the first toestimate the yield of the first atom bomb)

Emily Rosa the 11 yr old who waspublished in JAMA (youngest author ever)for her experiment that debunkedtherapeutic touch


6/23


7/23 Copyright 2007 Hubbard Decision Research and The Advisory Council, Inc. 7

Before We Measure We Ask:

Why do you want to know?

How much do you know now? What is the value to additional

information?



Why Do You Want To Know? The EPA needed to compute the ROI of the Safe Drinking WaterInformation System (SDWIS)

Why?: To prioritize three specific upgrades

We built a spreadsheet model that connected the expected effects ofthe system to relevant impacts

Source: Hubbard Decision Research



Uncertainty, Risk & Measurement Measuring Uncertainty, Risk and the Value of

Information are closely related concepts, importantmeasurements themselves, and precursors to mostother measurements

The Measurement Theory definition of measurement:A measurement is an observation that results ininformation (reduction of uncertainty) about aquantity.

We model uncertainty statistically

25% 50% 75% 100% 125%-25% 0%

Risk ofNegative ROI

Expected ROIROI = 0%

Return on Investment (ROI)



How Much Do You Know Now?

Most people can be taught how tosubjectively assess their current state of

uncertainty A small amount of training (three hours)can significantly improve calibration ofestimates

Comparisons of actual measures to original

calibrated estimates show calibration works



Risk/ROI w/ Monte Carlo

Administrative CostReduction

Total Project Cost

% Improvement inCustomer Retention

5% 10% 15%

10% 20% 30%

$2 million $4 million $6 million

ROI

-50% 50% 100%0%

Inputs



What Is The Value of Measurement?

Worst Bound of the 90%

CI; this is the undesirable

end of the range

Best Bound of the 90%

CI; this is the desirable

end of the range

Threshold: Below

this point is losingmoney

A

B Relative Threshold

(RT)=B/A

As a rule of thumb, the value of information is simplythe cost of being wrong times the chance of beingwrong

The value of information on a range often just comesdown to where the threshold is within the range.

Source: Howto Measure Anything: Finding the Value of Intangibles in Business
http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120



The IT(?) Measurement Inversion

TypicalAttentio

nEconomicRelev

ance

Receives MostAttention

Least Relevant toApproval Decisions

Receives LeastAttention

Most Relevant toApproval Decisions

Costs

Initial Development Costs

Ongoing Maintenance/Training

Costs

Benefits

A specific benefit (productivity,sales, etc.)

Utilization (when usage startsand how quickly usage grows)

Chance of Cancellation

See my article The IT Measurement Inversion in CIO magazine
http://www.cio.com/archive/enterprise/041599_checks.htmlhttp://www.cio.com/archive/enterprise/041599_checks.html



Next Step: Observations Now that we know what to measure

and what its worth to measure, we

can think of observations that wouldreduce uncertainty

The value of the information limitswhat methods we should use, but we

have a variety of methods available Take the Nike Method: Just Do It

dont let imagined difficulties get inthe way of starting observations



The Power of Sampling Methods Several clever sampling methods exist that

can measure more with less data than youmight think

Examples: estimating the population of fish inthe ocean, estimating the number of tankscreated by the Germans in WW II, extremelysmall samples, etc.




The Math-less Statistics Table Measurement is based on

observation and mostobservations are just

samples Reducing your uncertainty

with random samples is notmade intuitive in moststatistics texts

This table makes computing

a 90% confidence intervaleasy

Source: How to Measure Anything: Finding the Value ofIntangibles in Business
http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120http://www.amazon.com/gp/product/0470110120?ie=UTF8&tag=theadvisoryco-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0470110120



Measuring to the Threshold Measurements have

value usually

because there issome point wherethe quantity makesa difference

Its often muchharder to ask Howmuch is X than IsX enough




The Simplest Method

Bayesian methods in statistics usenew information to update prior

knowledge Bayesian methods can be even more

elaborate that other statisticalmethods BUT

It turns out that calibrated people arealready mostly subjectively Bayesian



Improving on Human Judgment The Lens Model is one method used to improve on expert intuition The chart shows the reduction in error from this method on

intuitive estimates In every case, this method equaled or bettered the judgment of

experts

0% 5% 10% 15% 20% 25% 30% 35% 40%

Cancer patient life-expectancy

Life-insurance salesrep performance

Graduate students grades

Changes in stock prices

Mental illness using personality tests

Student ratings of teaching effectiveness

IQ scores using Rorschach tests

Psychology course grades

Business failures using financial ratios

Reduction in ErrorsSource: Howto Measure Anything: Finding the Value of Intangibles in Business



Prediction Markets Simulated

trading marketsare a proven

method ofgeneratingprobabilities foruncertainevents

Research shows

that it workseven withoutpurelymonetaryreward systems Source: Servan-Schreiber, et. al. Electronic Markets, v14-3, September 2004



Applied Information Economics The basic AIE model

was used in several ofthese examples where

intangiblesdominated the decision Even though it has

been used on problemsas different as IT portfolio

prioritization Forecasting fuel for

the battlefield

the same basicprocess applies

Add/updatevalues in model

Select/Calibrateestimators

ComputeInformation

Values

Select/DesignMeasurement

Instrument

Do anyVariables havea significant

EVPI?

Collect/AnalyzeNew

Observations

Start: DefineDecision(s)

Decompose theVariable

(if possible)

Yes

No

MakeDecision

(Pseudo-Bayesian oradditionalexpertpolling)

Build adecision model

(Ongoing evaluationsdecisions and coursecorrections)

Source: Howto Measure Anything: Finding the Value of Intangibles inBusiness


22/23


Final Tips Learn how to think about uncertainty, risk

and information value in a quantitative way

Assume its been measured before

You have more data than you think and youneed less data than you think

Methods that reduce your uncertainty aremore economical than many managers

assume Dont let exception anxiety cause you to

avoid any observations at all

Just do it


23/23

More Information: Several of my articles in CIO magazine,

InformationWeekand other periodicals discussmeasurement - especially the risk and value of IT. Allof them are linked on www.hubbardresearch.com.

The book will have its own web site atwww.howtomeasureanything.com. The site will beonline before the book is released in July. It willinclude detailed examples and spreadsheetsmentioned in the book as well as discussion groups forreaders.

Contribute to The Measurement Challenge bysending us your examples of difficult measurements.
http://www.hubbardresearch.com/http://www.howtomeasureanything.com/http://www.howtomeasureanything.com/http://www.hubbardresearch.com/

tac how to measure anything

Documents