[mobiconf 2014] shazam mobile apps - data driven project management

Shazam mobile apps - Data Driven Project Management

Tomasz Kustrzynski, October 2014

© 2014 Shazam Entertainment

About Shazam

music recognition service and apps

founded in 1999

launched in the UK in 2002

one of the most popular mobile apps

500M+ downloads


Shazam technology - then

(phone you most likely used in 2002)

1. call 25802. play the music3. wait for SMS with results


Shazam technology - now

since 2008 - iPhone and Android appssince 2011 - integrated with Spotifysince 2014 - integrated with iOS8’s Sirimusic streaming, TV ads, live TV,and more


Shazam is now a part of popular culture...


...with lots of uses...


...but there’s still a lot to do

organisation

embrace the change

spiral development

why and what do we measure

estimation (and causation bias)

handling dependencies

Agenda

Organisation


The teams


Technical Project Management

the scientific method

hypothesis experimentanalyse

datadraw

conclusions

improve continuously(Kaizen)


We use Kanban

visualise workflow


We use Kanban

visualise workflow

limit Work In Progress(WIP)


We use Kanban

visualise workflow

limit Work In Progress

manage the flow


We use Kanban

visualise workflow


manage the flow

define explicit policies


We use Kanban

visualise workflow


manage the flow

define explicit policies

improve collaboratively


Why KanbanOur work is complex, requirements and circumstances change.

(fixed scope iterations won’t work)

We can be flexible with release cadence.

(we release when we’re ready)

Visibility and metrics.

(value stream mapped on the board)


This talk is not about theory

why limit work in progress?

why focus on finishing, not starting?

why slack is good for efficiency, ‘busyness’ is not?

why early feedback loops are necessary?

effectiveness vs efficiency?

Embrace the change


What do we care about

Users. We now have millions monthly active users.

100+


Types of work

new functionalities commercial deals

deadlines(quality still matters)

defined scope(flexible interpretation)

no deadlines(quality most important)

flexible scope(UX most important)


commercial dealsnew functionalities

Types of work

requirements change rapidly in both cases


Hack time (3rd type of work)

some time can be used to work on any project

Shazam for Mac was one of them


Hack time - results


Last minute changes


Real Options model


Real Options model

● options have value(you’ll have more information/make better decision later)

● options expire(ability to execute option ceases to exist at some point)

● never commit early unless you know why(sometimes no decision is the best decision)


Real Options in practice

● no long queues● designs and specs refined as we go● A/B testing● shorter release cycles● irreversible commitments


Real Options - exampleearly commitment

backlog too long

backing out of early commitment

late commitment

Spiral development


Ceremonies

ceremonies limited to a minimum

meetings often ad hoc, here and now

● morning standups● retrospectives● visual reviews

● bug triage meetings● detailed planning● estimation


Iterative/spiral development1. update requirements2. implement3. visually review new

functionalities(usually work in progress)

4. collect feedback5. repeat until signed off


Visual reviewsin the breakout area for everyone to join


Visual reviews

in the breakout Terminal 6 area for everyone to join

Why and what do we measure?


T6 - taking the metaphor further

flight board

shows status of epics

all teams included

we track more than this...


Data driven decisions


Data gathering

"In God we trust; all others must bring data."Edward Deming

"What's measured improves."Peter F. Drucker


Data gathering

information radiators across the office

Estimation


Estimates are dangerous


Estimation scale we use

1

TFB - too f* big

NFC - no f* clue

[give me a minute]

cards in the photo by

Lunar Logic (@Lunar_Logic)

FAQ 1

As a Product Manager, when can I expect this Story to be completed?


Disney/Cycle Times

time for a Story to get implemented and tested(or time from entering queue until finishing the ride in Disneyland)

having lots of samples we can use statistics(and talk about probabilities)


Disney/Cycle Times

parallel streams of

work

Cycle Times

t

start of development

start of test

work done


Disney/Cycle Times - statistics

histogram of Disney Times (Cycle Times) density function

50th, 85th percentile(50/85% items get finished within this time)


When a Story will get done?

1. You can get this story within X days with 85% probability.

2. Probability of you getting it within a week is Y%.


Scientific method in actionwe track median (50th percentile) Cycle Times:

tracking metrics in time allows us to use scientific method


Hypothesis

limiting Work In Progress will shorten Cycle Times

according to Little’s Law:

avg Cycle Time = avg Work In Progressavg Throughput


Correlation

Work In Progress limit for Development lowered

correlated results next day


Does eating ice cream make you more prone to drowning?


Causation?we had a glitch in the system, but over the next month results matched our hypothesis, and we could replicate the experiment

Ok, so when can we get all these 10 stories done?

FAQ 2


it’s easier with Takt TimesTakt Times

Modelling parallel work

parallel streams of

work

parallel work is difficult to model with cycle times

Cycle Times

t


Distribution of average Takt Times

create set of average Takt Time samples based on similar project

resample and ∑many times

(>1000)

average

distribution of average Takt Timesaverage Takt Times

Takt Times


Predicting delivery time for N items

average TT

* Nrepeat many

times (>1000)average TT

average TT

average TT

distribution of expected delivery timesaverageTakt Times

expected delivery times

85% probability to deliver within this time


When will all the Stories get done?

1. You can get these Stories within X days with 85% probability.

2. Probability of you getting them within a week is Y%.

FAQ 3

Why did we end up with large number of stories to test at once?

Do some types of items take longer to develop than others?

Are there times when teams are more productive?

Did we split Stories properly or we missed splitting some big ones?


Patterns and improvement

Being aware of patterns enables us to act on them:

● introduce better policies(e.g. all UI assets for a Story done before dev commences)

● change development cycle(e.g. release more often to prevent batching)

● treat some types of work differently(e.g. allow extra dev time)


Control chart


Control chartweekends

Dependencies and blockersAs a Project Manager, what should I focus on next?


Blockers/dependencies

● visualise● prioritise

(what is blocked, for how long)● act● collect statistics and review them

(have any recent changes to the process change amount of blocked items/how long they’re blocked for?)


What should I do first? - answers

Team A: finally doing well

Team B: keeps getting blocked4 blockers including 1 old

Team C: doing well

Team D: 1 new blocker


Blockers statistics

trend of increasing median time to unblock

Cumulative Density Function(80% blocked items unblocked in ~3 days)

● use Kanban for flexibility, continuous improvement● embrace change, use Real Options rules● daily visual reviews allow fast feedback cycle● make decisions based on data● make estimates based on statistical analysis● handle dependencies and blockers

Summary

Thank [email protected]


References

● Real Options: commitment-thebook.com, Olav Maassen, Chris Matts● Real Options: http://www.infoq.com/articles/real-options-enhance-agility● Real Options: http://decision-coach.com/lean-and-real-options/● “High level planning using Monte Carlo simulation”, Dimitar Bakardzhiev● “Thinking, fast and slow”, Daniel Kahneman● http://en.wikipedia.org/wiki/Takt_time

[mobiconf 2014] shazam mobile apps - data driven project management

Leadership & Management