metrics 3 - andycleff.com · metrics 3.0 • andy cleff • @justsitthere 12 rules for measurement...

Metrics 3.0

2017 Mile High Agile - DenverPresented by Andy Cleff

Co-Author Ralph van RosmalenBased on work of Jurgen Appelo

Metrics 3.0 • Andy Cleff • @JustSitThere

Overview

Who’s Here Today?

Twelve Rules for Measurement

Group Brainstorming

Group Presentations

Keeping in Touch


12 Rules for Measurement

1: Measure for a purpose

2: Shrink the unknown

3. Seek to improve

4: Delight all stakeholders

5: Distrust all numbers

6: Set imprecise targets

7: Own your metrics

8: Don’t connect metrics to rewards

9: Promote values and transparency

10: Visualize and humanize

11: Measure early and often

12: Try something else

change and innovation practicesMANAGEMENT 3.0


When selecting metrics, ask:

Rule 1: Measure for a purposeYou must always understand why you are measuring. The metric is not a goal in itself. Never forget that it’s just a means to an end. It all starts with why.

Rule 2: Shrink the unknownA metric is just a surrogate for what you really want to know. Don’t jump to conclusions. Always try to reduce the size of what is still unknown.

Rule 3. Seek to improveDon’t only measure things that will make you look good. There is plenty of data around, but you must focus on what enables you to do better work.

Rule 4: Delight all stakeholdersYour work depends on others, and others depend on you. Never optimize for just one stakeholder. Instead, measure your work from multiple perspectives.

Rule 5: Distrust all numbersObservers usually influence their own metrics, and they suffer from all kinds of biases. Have a healthy, skeptical attitude towards any reported numbers.

Rule 6: Set imprecise targetsWhen people have targets, they have an inclination to focus on the targets instead of the real purpose. Avoid this tendency by keeping your targets vague.

Rule 7: Own your metricsEveryone is responsible for their own work, and metrics help us improve that work. Therefore, everyone should be responsible for their own metrics.

Rule 8: Don’t connect metrics to rewardsRewards often kill intrinsic motivation and lead to dysfunctional behaviors in organizations. Don’t incentivize people to do work they should like doing.

Rule 9: Promote values and transparencyHuman beings are smart and able to game any system. To prevent gaming, be transparent about values, intentions, and the metrics everyone is using.

Rule 10: Visualize and humanizeNumbers tend to dehumanize everything. Replace digits with colors and pictures, and keep the measurements close to where the actual work is done.

Rule 11: Measure early and oftenMost people don’t measure often enough. Measure sooner and faster to prevent risks and problems from growing too big for you to handle.

Rule 12: Try something elseIt’s rarely a good idea to do the same things over and over. The environment changes all the time. The same should apply to how and what you measure.

› Why “this metric?” – Why does it matter?

› What insights might we gain from it?

› What is expected to change? What is expected variability, consistency – are we looking for trends or absolute values?

› How might it be gamed, misused (or abused)?

› What are some for trade offs / costs of improvement - Working to improve one thing may temporarily reduce another (e.g., predictability may increase at the expense of throughput)

› How often would we like to “take a data point”?

› How long will we run the experiment? (What is the half-life?)

› How when we know when we’re “done” with this metric (and it’s served its purpose, and it’s time to retire it and consider another…)?

› How will we make our measurements transparent – to promote knowledge sharing, collaboration with other teams and trust with our sponsors?

› Is this metric a leading or lagging indicator?

Rule 1- Measure for a purpose

You must always understand what you are measuring. The metric is not a goal in itself. Never forget that it’s just a means to an end. It all starts with why.

“If all we have are opinions, let’s go with mine.”

Jim Barksdale

“…Analysis without numbers is only an opinion.”

Atkins Law #1


Reasons why we do measure

To see revenues to drive resource & people allocation

Monitor alignment with mission / vision / goal

Observe quality of product / process

Judge customer happiness / employee satisfaction

To make decisions that are not based on gut feelings


Reasons why we don’t measure

Measurements might be used as weapons

Lame metrics that would not useful or actionable

Implementing measures would cost too much time / effort

Some things might just not be immeasurable

Rule 2 - Shrink the unknown

A metric is just a surrogate for what you really want to know. Don’t jump to conclusions. Always try to reduce the size of what is still unknown.


Cynefin Framework


Cynefin FrameworkOrdered Domains


Cynefin Framework

Here Things Get Interesting…

Rule 3 - Seek to improve

Don’t only measure things that will make you look and feel good. There is plenty of data around, but you must focus on what enables you to do better work.


Actionable Metrics

“A good metric changes the way you behave. This is by far the most important criterion for a metric: what will you do differently based on changes in the metric?”

Lean Analytics, Alistair Croll and Benjamin Yoskovitz


Vanity Metrics

“When we rely on vanity metrics, a funny thing happens. When the numbers go up, I've personally witnessed everyone in the company naturally attributing that rise to whatever they were working on at the time. That's not too bad, except for this correlate: when the numbers go down, we invariably blame someone else”

Eric Ries

Rule 4 - Delight all stakeholders

Your work depends on others, and others depend on you. Never optimize for just one stakeholder. Instead, measure your work from multiple perspectives.


It is impossible to please everyone, but you would like to know who is pleased at certain moments and who is not.

Rule 5 - Distrust all numbers

Observers usually influence their own metrics, and they suffer from all kinds of biases. Have a healthy, skeptical attitude towards any reported numbers.

Story

Hawthorne Works, Chicago, ca. 1925.

Rule 6 - Set imprecise targets

When people have targets, they have an inclination to focus on the targets instead of the real purpose. Avoid this tendency by keeping your targets vague.

"When a measure becomes a target, it ceases to be a good measure."

Goodhart's law

Rule 7 - Own your metrics

Everyone is responsible for their own work, and metrics help us improve that work. Therefore, everyone should be responsible for their own metrics.


Important Considerations

How many metrics should a team use?

Which ones to use?

How long should they use the ones selected?


Anti-Patterns

Looking at a single metric (Hawthorn)

Striving for ever increasing values instead of striving for consistency and stability (Goodhart)

Correlation is not necessarily causation (Milton Friedman’s Thermostat)

Comparing metrics across teams that are very different

Rule 8 - Don’t connect metrics to rewards

Rewards often kill intrinsic motivation and lead to dysfunctional behaviors in organizations. Don’t incentivize people to do work they should like doing.

Rule 9 - Promote values and transparency

Human beings are smart and able to game any system. To prevent gaming, be transparent about values, intentions, and the metrics everyone is using.


Values, Intention, Purpose

Do we get paid a sustainable value for what we do?

Are we great at what we do in the eyes of our customers?

Do our employees / team mates love what we do and the way we do it?

Will what we do make the world a better place for our grandchildren?

Rule 10 - Visualize and humanize

Numbers tend to dehumanize everything. Replace digits with colors and pictures, and keep the measurements close to where the actual work is done.

Rule 11 - Measure early and often

Most people don’t measure often enough. Measure sooner and faster to prevent risks and problems from growing too big for you to handle.

“The only way to win is to learn faster than anyone else”

Eric Ries

“What you want to do as a company is maximize the number of experiments you can do per unit of time.”

Jeff Bezos

Rule 12 - Try something else

It’s rarely a good idea to do the same things over and over. The environment changes all the time. The same should apply to how and what you measure.


Limited Lifespan of all Metrics

That which is measured will improve, at a cost.

When a measure becomes a target, it ceases to be a good measure.

Correlation is not causation, but it sure is a hint.

Use multiple viewpoints - technical as well as human - to get a holistic perspective


The Twelve Rules for Metrics

1: Measure for a purpose

2: Shrink the unknown

3. Seek to improve

4: Delight all stakeholders

5: Distrust all numbers

6: Set imprecise targets

7: Own your metrics

8: Don’t connect metrics to rewards

9: Promote values and transparency

10: Visualize and humanize

11: Measure early and often

12: Try something else

Collective Brainstorming


Five Categories of Metrics

1. Process Health Metrics - assess day-to-day delivery team activities and evaluates process changes.

2. Release Metrics - focus on identifying impediments to continuous delivery.

3. Product Development Metrics - help measure alignment of product features to user needs.

4. Technical / Code Metrics - help determine quality of implementation and architecture.

5. People/Team - reveal issues that impact a team’s sustainable pace and level of engagement.

Shout out to Jason Tice @theagilefactor

How to Choose?


Review the Options….

1. Why “this metric?” – Why does it matter? Who does it matter to?

2. What insights might we gain from it?

3. What is expected to change? Are we looking for variability, consistency, trends or absolute values?

4. How might it be gamed, misused (or abused)?

5. What are some trade offs / costs of improvement?

6. How often would we like to “take a data point”?

7. How long will we run the experiment?

8. How when we know when we’re “done” with this metric?

9. How will we make our measurements transparent?

10.Is this metric a leading or lagging indicator?

Debrief / Preso’s

Let’s keep the conversation going…

Andy [email protected]/in/[email protected]

http://coalition.agileuprising.com

40+ Metrics for Software TeamsPeople/Team: Human ElementsThis group of metrics reveals issues that impact a team’s sustainable place and level of engagement.

› Team Happiness / Morale / Mood › Gallop Q12 › Team / Manager / Organization NPS › Percentage of time w/o interruptions › Trust between Leadership and Team › Learning Log › Team Tenure › Phone-a-Friend Stats › Whole Team Contribution › Transparency (access to data, access to customers,

sharing of learning, successes and failures) › Comparative Agility: Team mapping against the 12 agile

principles (Geoff Watt’s “Scrum Mastery”)

Process Health MetricsThis category assess day-to-day delivery team activities and evaluates process changes.

› Cumulative Flow Diagrams › Control Charts › Cycle Time › Percent Complete and Accurate › Time Blocked per Work Item › Story/Epic Lead Time › Successful Iteration Completion › Escaped Defect Resolution Time

Release MetricsThis group directs focus on identifying impediments to continuous delivery.

› Escaped Defects › Release Success Rate › Release Time › Time Since Last Release › Cost Per Release › Release Net Promoter Score › Release Adoption / Install Rate

Product Development MetricsThese help measure alignment of product features to user needs.

› Customer / Business Value Delivered › Risk Burndown › Value Stream Mapping › Sales Velocity › Product Forecast › Product Net Promoter Score (NPS) › User Analytics

Technical/Code MetricsThe following help determine quality of implementation and architecture.

› Test Coverage › Unit/Regression Test Coverage › Build Time › Defect Density › Code Churn › Code Ownership › Code Complexity › Coding Standards Adherence › Crash Rate › Build Breaks › Technical Debt › Ratio of Fixing Work vs Feature Work

Andy CleffAndy is an experienced and pragmatic agile practitioner that takes teams beyond getting agile to embracing agile. His chief weapons are well asked questions, insightful retrospectives and an ability to withstand awkward silences. And if all else fails, beer.

[email protected]

andycleff.com

linkedin.com/in/andycleff

@JustSitThere

agileuprising.com

The following listing is intended as a starting point for conversation and discussion. Choose one or two that make sense for your team / organization and add them to your current dashboard. Then rinse and repeat over time.

change and innovation practicesMANAGEMENT 3.0


When selecting metrics, ask:

Rule 1: Measure for a purposeYou must always understand why you are measuring. The metric is not a goal in itself. Never forget that it’s just a means to an end. It all starts with why.

Rule 2: Shrink the unknownA metric is just a surrogate for what you really want to know. Don’t jump to conclusions. Always try to reduce the size of what is still unknown.

Rule 3. Seek to improveDon’t only measure things that will make you look good. There is plenty of data around, but you must focus on what enables you to do better work.

Rule 4: Delight all stakeholdersYour work depends on others, and others depend on you. Never optimize for just one stakeholder. Instead, measure your work from multiple perspectives.

Rule 5: Distrust all numbersObservers usually influence their own metrics, and they suffer from all kinds of biases. Have a healthy, skeptical attitude towards any reported numbers.

Rule 6: Set imprecise targetsWhen people have targets, they have an inclination to focus on the targets instead of the real purpose. Avoid this tendency by keeping your targets vague.

Rule 7: Own your metricsEveryone is responsible for their own work, and metrics help us improve that work. Therefore, everyone should be responsible for their own metrics.

Rule 8: Don’t connect metrics to rewardsRewards often kill intrinsic motivation and lead to dysfunctional behaviors in organizations. Don’t incentivize people to do work they should like doing.

Rule 9: Promote values and transparencyHuman beings are smart and able to game any system. To prevent gaming, be transparent about values, intentions, and the metrics everyone is using.

Rule 10: Visualize and humanizeNumbers tend to dehumanize everything. Replace digits with colors and pictures, and keep the measurements close to where the actual work is done.

Rule 11: Measure early and oftenMost people don’t measure often enough. Measure sooner and faster to prevent risks and problems from growing too big for you to handle.

Rule 12: Try something elseIt’s rarely a good idea to do the same things over and over. The environment changes all the time. The same should apply to how and what you measure.

› Why “this metric?” – Why does it matter? Who does it matter to?

› What insights might we gain from it?

› What is expected to change? What is the expected variability and consistency – are we looking for trends or absolute values?

› How might it be gamed, misused (or abused)?

› What are some for trade offs / costs of improvement?

› How often would we like to “take a data point”?

› How long will we run the experiment? (What is the half-life?)

› How when we know when we’re “done” with this metric?

› Are we adding to the dashboard or replacin/retiring something else?

› How will we make our measurements transparent – to promote knowledge sharing, collaboration with other teams and trust with our sponsors?

› Is this metric a leading or lagging indicator?

Metrics 3.0 • Andy Cleff • @JustSitThere • AndyCleff.com

Good Ideas / Actionable Metrics

› Team / Manager / Organization NPS

› Gallop Q12

Bad / Nasty / Vanity Metrics

› Time to hire

› Lines of code per individual

People/Team: Human ElementsThis group of metrics reveals issues that impact a team’s sustainable pace and level of engagement.


Process Health MetricsGood Ideas / Actionable Metrics

› Cumulative Flow Diagrams

› Lead & Cycle Time


› Velocity

› Story points per developer

This category assess day-to-day delivery team activities and evaluates process changes.


Release MetricsGood Ideas / Actionable Metrics

› Time to first release

› Frequency of release


› Lines of code pushed

› Story points per release

This group directs focus on identifying impediments to continuous delivery.


Product Development MetricsGood Ideas / Actionable Metrics

› Product Net Promoter Score (NPS)

› Risk Burndown


› Number of new features

› Customer satisfaction survey conducted by sales agent

These help measure alignment of product features to user needs.


Technical/Code MetricsGood Ideas / Actionable Metrics

› Ratio of Fixing Work vs Feature Work

› Test coverage


› Lines of code

› Causal Analysis

The following help determine quality of implementation and architecture.

metrics 3 - andycleff.com · metrics 3.0 • andy cleff • @justsitthere 12 rules for measurement...

Documents