Metrics for the LEAN/Agile Product Manager

Yuval Yeret

Main things we want to pay attention to

• Performance of the Production floor – covered elsewhere (Simple KPIs Slides)

• Performance of the Product Management group: – Business Value–Wastes related to PM– Technical Debt

Business value

• We care about outcome – features delivered, adopted, used, paid for

• How can we measure this? • Manage a kanban at the high level features level,

that tracks when features are adopted, and upon first paying customer.

• Then see how much WIP of features not yet adopted we have, LEAd/Cycle time to adoption, features that we dropped on the way.

Debt

• A lot of time debt is taken due to PM decision

• We want to track how much debt we have, and take action to minimize it.

• E.g. we need to release CustomerFeatureX now, so we don’t “automate tests”/”code it correctly”, so every work on ModuleY which is used in FeatureX is slower, until this is fixed

Tracking debt in kanban

• Have debt card type that is created when debt is taken on

• Track amount of debt versus overall WIP/Backlog

• See whether stable, improving, worsening trend

• Decide on policy for dealing with debt – WIP Limit, etc.

• Track the cycle time and WIP for debt cards to see whether they get the SLA they deserve

Wastes related to PM

• Waiting for PM• PM related Churn / Context switching /

Expediting• Sunk Costs• Rework due to late feedback by PM

Waiting for PM

• Look at the CFD, observe the size of the PM-related queues over time. – Especially Pending PM Review which is in the

middle of DEV/Test– And Ready-MMFs as well as DEV Ready in some

cases which depend on PM approval– Advanced – in the cycle time performance

report, focus on PM areas

• When looking at exceptions to Cycle time, participate in the root cause analysis, and see if interaction with PM was part of the long cycle time.

PM related Churn / Context switching / Expediting

• Add Expedited class of service– Can be used by PM to

override priorities in DEV WIP – just to top of queue, don’t override current WIP

• Add emergency class of service– Can also override current

WIP

• Assumption – – This is value trumps flow.

We give up efficiency when we use these COSs

Measuring the effect of value over flow COS

• Look at cycle times for different kinds of classes of service

• Look at distribution of different COS in the WIP

Look at amount of changes in scope

• Replace – need visualization that shows scope changes in content

• Add – can simply look at total scope for a “Release” and observe whether its growing

Case Study – Typical release behavior

1 2 3 4 5 60

20

40

60

80

100

120

140

160

180

200

Total Required WorkLinear (Total Required Work)Actual DonePlanned ScopePlanned DoneOriginal + Dark MatterLinear (Original + Dark Matter)

Added Scope

Growth in Feature

Cost / Dark Matter

• Dark Matter – Is where we thought a feature costs X

• But then, during breakdown, analysis, creating iteration stories, we understand it actually costs X+D

• Then, PM decides whether to scope to fit down to X again, or D is worth it.

• Worthwhile tracking our behaviour on this, and learning from it.

• What is the right D number/percent? Good question!

• Can be observed in the CFD for a release.

Sunk Costs

• Add a LANE that collects features/stories that are “ON HOLD” – the Recycle Bin in the archive area

• The amount of work done on them is the sunk cost

• Amount of work hard to measure, so use alternative:– CYCLE TIME – look at cycle time for end

lane being the recycle bin

Rework due to late feedback by PM

• Will appear as high cycle times• Will appear as moving back cards on

the board (need to find way to measure)

• Can use special Class of service / card type to identify these kinds of stories better for measurement/tracking purposes

Workload compared to DEV

• See how much workload is in PM compared to DEV

• Look for trends and major changes that can indicate:– Bottleneck in PM– Idle and slack capacity – expect to see PM seeing

clients/customers at those times

Release OVERHEAD

• “How often do we release? What does it cost us?”

• The Usual Suspects– PM wants to release more often. He wants

the ready feature to be out there as soon as possible.

– R&D usually wants to limit the amount of releases, as they cost a lot, and R&D doesn’t like to do hardening

How often do we release

• On kanban, simply add a card type of “Release” and flow it thru the board to signify releases.

• The size should be the hardening cost planned for this release.

• based on SPC and other charts, you can understand:– Plan versus actual on hardening costs/times/dates– Frequency of actual releases– Ratio of hardening work compared to feature work

(see next slide for a view on this)

V3.0 V3.5 V4.0 V4.50

20

40

60

80

100

120

Indirect/Other Ef-fort

Effort on Stabi-lization

Effort on New Features

Release

% o

f R

ele

ase B

ud

get/

Eff

ort

Release Overhead

• This metric shows how much effort is spent on releasing versus developing.

• The aim is to reduce the overhead of each release, such that the organization can increase the frequency of releases to meet business expectations.

Reducing the release overhead

• Two things we need to do:– Reduce the overhead of each release–Make sure our release frequency makes

sense

Reducing the release overhead of each release

• Invest in reducing legacy hardening debt– As the PM you’ll be asked to invest. – Ask for a plan that associates investment of X days with Y days

of reduction in hardening cost. – Decide what is your investment horizon– Based on the horizon, the X/Y ratio, and the current frequency of

releases, make your decision– Typical areas of investment - Continuous Integration, Automation

of EVERYTHING (Including platform matrix, performance, any thing that is currently in the hardening plan)

• Avoid hardening debt while developing new features• Build quality in – don’t let defects wait for the end• Consider different types of releases – e.g. Majors,

Feature/Service Packs, Patch Bundles– And associate the relevant risk-driven hardening work for each

Does our release frequency make sense?

• First step – have the visibility– How many releases– What kinds of releases – scheduled major

“Trains”, emergency fixes “Ambulance?”, “special delivery/Taxis”

• Second step – lay out what the business actually needs and is willing to pay for

• Are those aligned?• A lot of the times, you will see “Trains” and

“Taxis”. Think of investing in a “Subway” – a predictable frequent release mechanism