agile metrics: make better decisions with data

K4 Keynote 11/17/2016 4:15:00 PM

Agile Metrics: Make Better Decisions with Data

Presented by:

Larry Maccherone

AgileCraft

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073

888-‐268-‐8770 ·∙ 904-‐278-‐0524 - [email protected] - http://www.stareast.techwell.com/

mailto:[email protected]

http://www.stareast.techwell.com/

Larry Maccherone AgileCraft

An industry-recognized leader in agile, metrics, and visualization, Larry Maccherone currently helps a number of companies with the design of their analytics products including AgileCraft and Pendo.io. Previously, Larry led the insights product line at Rally Software which enabled better decisions with data, leveraged big data techniques to conduct groundbreaking research, and offered the first-ever agile performance benchmarking capability. Before Rally, Larry worked at the Software Engineering Institute for seven years conducting research on software engineering metrics with a particular focus on reintroducing quantitative insight back into the agile communities.

11/3/2016

1

Make BetterMake Better Decisions with Data

Larry MaccheroneLinkedIn.com/in/LarryMaccherone

LinkedIn.com/in/LarryMaccherone


11/3/2016

2

What?What?So what?

NOW WHAT?


What?


11/3/2016

3


So what?


11/3/2016

4

Visualization is like photography. Impact is a function of focus, illumination, and perspective.

What?


Credit: Edward Tufte

NOW WHAT?


11/3/2016

5

D ’tDon’tLaunch!


Prevent your own disastrous decisions


11/3/2016

6

Larry MaccheroneLinkedIn.com/in/LarryMaccherone


Cognitive biasworks against good decisions


11/3/2016

7

We don't see things the way they are.

We see things the way we are.

LinkedIn.com/in/LarryMaccherone LinkedIn.com/in/LarryMaccherone

~The Talmud

Next slide is a movieclick to play


11/3/2016

8

Denying the Evidence


Truths about cognitive bias

1. Very few people are immune to it.

2 We all think that we are part of that2. We all think that we are part of that small group.

3. You can be trained to get much, much better. Douglass Hubbard – How to Measure Anything

4. We do a first-fit pattern match. Not a best-fit pattern match And we only use about 5% of the information


match. And we only use about 5% of the information to do the matching.

5. We evolved to be this way (survival trait).

11/3/2016

9



11/3/2016

10

An example of overcoming cognitive bias


We are overconfident when assessing our own uncertainty

But, training can “calibrate” people so that of all the times they say they are X% confident, they will be right X% of the time

Trained/Calibrated

Untrained/Uncalibrated

Statistical Error

“Ideal” Confidence

50%

60%

70%

80%

90%

100%

75 71 65 58

21

17

68 15265

4521

Per

cent

Cor

rect

# f R


30%

40%

50% 60% 80% 90% 100%

25

70%

Assessed Chance Of Being Correct

P

99 # of Responses

Copyright HDR 2007 [email protected]

11/3/2016

11

Equivalent Bet calibration

What year did Newton published the Universal Laws of Gravitation?

Pick year range that you are 90% certain it would fall within.

Win $1,000:

1. It is within your range; or

2. You spin this wheel and it lands green

10%


Adjust your range until 1 and 2 seem equal.

Even pretending to bet money works.

90%

Agile Teams Programs andAgile Teams, Programs, and Portfolios

benefit from similarcalibration exercises


11/3/2016

12

Every decision is aforecast!


You are forecasting that gyour choice will have better

outcomes than the other alternatives


alternatives

11/3/2016

13

How to avoid cognitive bias in decision making

Don't focus on consensus. Ritual dissent is a much more successful approach.

“But that doesn’t explain _______”. An FBI agent knew that some folks were being trained to fly but not take off and land.

Assign someone the role of devil’s advocate


Assign someone the role of devil s advocate.Israel’s 10th man.

In other words… Really consider the other ALTERNATIVES

Types of bias

http://srconstantin.wordpress.com/2014/06/09/do-rationalists-exist/


11/3/2016

14


For a given alternative, let:Pg = Probability of good thing happeningV = “Value” of good thing happening


Vg = “Value” of good thing happening

Then:Value of the alternative = Pg × Vg

11/3/2016

15

AA lean/agile product

management example


$8M

Best case (25%)

Likely case (50%)

Worst case (25%)

1PW × VW = .25 × -$1.00M = -$0.25M

PL × VL = .50 × $1.00M = $0.50MP × V 25 × $8 00M $2 00M

$1M$1M

1

$2M$2M$1M2

PB × VB = .25 × $8.00M = $2.00M -----------$2.25M

PW × VW = .25 × $1.00M = $0.25MPL × VL = .50 × $2.00M = $1.00MPB × VB = .25 × $2.00M = $0.50M

-----------


Which strategy is best……for your company?…for your career?

$1.75M

11/3/2016

16

If you get only 1 project then strategy 2 is better

75% of the time

If you get ∞ projects thenstrategy 1 is better100% of the time


How many projects do you need for strategy 1 to be better more often than not?


11/3/2016

17

Play with it yourself at:http://jsfiddle.net/lmaccherone/j3wh61r7/



11/3/2016

18

Emotion and bias plays a part


Did any of you get emotional about the $1M loss?

Did any of you want to question the $8M number?

It’s critical to


It s critical to… …eliminate fear from the equation

…change the nature of the conversation

11/3/2016

19

Argument is about who is right.Decision making is about what is right.


AAnagile delivery date forecast

example


11/3/2016

20

Monte Carlo ForecastingWhat it looks likeLive demo: http://lumenize.com (use Chrome)


Seek toSeek tochange the nature of

the conversation


the conversation

11/3/2016

21

Criteria forgreat visualizationg

Credits:Edward Tufte (mostly)


Edward Tufte (mostly) Stephen Few

Gestalt School of Psychology

1. Answers the question, "Compared with what?” (So what?)


Trends

Benchmarks

11/3/2016

22

2. Shows causality, or is at least informed by it

The primary chart used by the NASA scientists showed O-ring failure indicators by launch datefailure indicators by launch date.

Tufte's alternative shows the same data by the critical factor, temperature.

The fateful shuttle launch occurred at 31 degree Tufte's visualization


at 31 degree. Tufte s visualization makes it obvious that there is great risk for any launch at temperatures below 66 degrees.

3. Tells a story with whatever it takes

Still

Moving

Numbers

Graphics

And …


Maybe some fun

11/3/2016

23

4. Is credible

Calculations explained

Sources

Assumptions

Who (name drop?)

Drill-down


How?

Etc.

5. Has business value(or value in it’s social context)

like Vic Basili’sGoal-Question-Metric (GQM)

but withoutISO/IEC 15939 baggage

The ODIM framework

D

I N S I G H T

M E A S U R E

THINK

EFFECT


O U T C O M E

D E C I S I O NTHINK

11/3/2016

24

6. Shows comparisons easily (1)

aka: Save the “pie” for dessert Credit:

• Stephen Few (Perceptual Edge)• http://www.perceptualedge.com/ar

ticles/08-21-07.pdf



Can you compare the market share from one year to the next?

Q i kl Whi h t i i h th f t t?Quickly: Which two companies are growing share the fastest?


One pie chart is bad. Multiple pie charts are worse!!!

11/3/2016

25


How about now?

Can you compare the market share from one year to the next?


7. Allows you to see the forestANDAND the trees


11/3/2016

26

8. Informs along multiple dimensions (1)


9. Leaves in the numbers wherewhere possible


11/3/2016

27

10. Leaves out glitter

Examples of how NOT to do it.


Top 10 criteria for great visualization

1. Answers the question, "Compared with what?”

6. Shows differences

Credits:• Edward Tufte• Stephen Few• Gestalt

(School of Psychology)

(SO What?)

2. Shows causality, or is at least informed by it. (NOW WHAT?)

3. Tells a story with whatever it takes.

easily.

7. Allows you to see the forest AND the trees.

8. Informs along multiple dimensions.

9 Leaves in the numbers where


takes.

4. Is credible.

5. Has business value or impact in its social context.

9. Leaves in the numbers where possible.

10. Leaves out glitter.

11. Uses good visual grammar

11/3/2016

28

“They” say…

Nobody knows what’s gonna happen next: not on a freeway, not in an

airplane, not inside our own bodies d t i l t t k ithand certainly not on a racetrack with 40 other infantile egomaniacs.

– Days of Thunder

Trying to predict the future is like trying to drive down a country road at night with no lights while looking

t th b k i d


out the back window. – Peter Drucker

Never make predictions, especially about the future.– Casey Stengel

When you come to a fork in the road…

take it!

~Yogi Berra

What?the metrics and analysis

S h t?


So what?how it compares/trendswhat it means

NOW WHAT?every decision is a forecast

11/3/2016

29

Questions?
