agile analytics: a gqm approach to enterprise metrics
TRANSCRIPT
© 2012 Fiserv, Inc. or its affiliates.
The GQM Approach to Enterprise Agile Metrics
Version 1.42
© 2012 Fiserv, Inc. or its affiliates.
JohnTannerSVP&EnterpriseTransformationConsultant,LeadingAgile
Ø 20+yearsinSoftwareDevelopmentØ RolesinindustryasConsultant,Engineer,
Architect,Product&ProjectManagerØ ImplementedAgileinalltheenvironments:
o Non-profito Corporateo Government
[email protected] www.linkedin.com/in/tannerjs www.facebook.com/leadingagile www.twitter.com/leadingagile
© 2012 Fiserv, Inc. or its affiliates.
Defining the Problem
© 2012 Fiserv, Inc. or its affiliates.
Threeviewsonmetrics
• Weneedmaximummetrics,becausewejustdon’tknowwhatwedon’tknow
• Weneedminimalmetrics,becausetheyjustdon’taddvalue
• Weneedsomemetrics,becausesomethingisbetterthannothing
© 2012 Fiserv, Inc. or its affiliates.
Weneedall ofthemetrics!
“Metricsarehowwemeasuresuccess.Solet’scapturemetricsforeverything.”
• Metricsdashboardsbecomeprolific,andsomeonehastomaintainthem
• EachTeamwindsupwithauniquesetofmetricstocapture,butwithnocommonexpectationforusingthem
• Theorganizationspendsmoretimecreatingmetricsroll-upsthantheydocreatingproduct
• Thereisnocohesivestrategyfortyingthedatatogethertomaketargetedimprovements
© 2012 Fiserv, Inc. or its affiliates.
Weneednone ofthemetrics!
“Metricsdonotaddvalue,sotheyarewaste.Let’seliminatethemaltogether.”
• Assumesallteamsarecurrentlyproducingworking,testedproductathighlevelsofefficiency
• Onlyworkswhentheorganizationdoesn’tcareiftheyeverimprovethesystem…
• …doesn’tcareiftheyaren’tproducingvalue• …doesn’tgethungupondetailslikemakingpayroll
© 2012 Fiserv, Inc. or its affiliates.
Weneedsomemetrics…
“Metricsmighthelpusimprove,soweneedmeasuresofsomesort.”
• Withoutclarity,weoftencapturemetricsformetrics’sake• Whenwegathermetricswithoutclearexpectationsaroundtheirusetheycanbewasteful,orevenharmful
• Wefocusonmetricsinawaythatultimatelydrives“localoptimization”attheexpenseoftheoverallsystem
• Weforgetthatthe“why”andthe“how”matters
© 2012 Fiserv, Inc. or its affiliates.
The “why” and the “how”
© 2012 Fiserv, Inc. or its affiliates.
Let’stalkaboutwhy,first…
Backlogitem
Backlogitem
Backlogitem
Backlogitem
Backlogitem
Backlogs
Team
Teams
WorkingTestedProduct
WorkingTestedProduct
© 2012 Fiserv, Inc. or its affiliates.
Howweoftenapproachtheproblem
InAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…
...isMakingandMeetingCommitments
...isProducingQualitySoftware
TheTeam...hasaStableVelocity
© 2012 Fiserv, Inc. or its affiliates.
Team
Backlogitem
Backlogitem
Backlogitem
Backlogitem
Backlogitem
WorkingTestedProduct
Makes&MeetsCommitments
Metric:StoryPoint
Completion%
HasaStableVelocity
Metric:VelocityVariance
ProducesQualitySoftware
Metric:EscapedDefect
Count
Howweoftenapproachtheproblem
© 2012 Fiserv, Inc. or its affiliates.
ProblemSolved?
Yay!Wediditteam!
No,butseriously,manypeoplethinklikethat…
© 2012 Fiserv, Inc. or its affiliates.
Howdoesthatworkatscale?
Team Team Team Team
TeamTeamTeam
Delivery Tier
Team Team TeamProgramTier
TeamPortfolioTier
© 2012 Fiserv, Inc. or its affiliates.
Goal-Question-Metric
© 2012 Fiserv, Inc. or its affiliates.
WhatisGQM?
TheGoal-Question-Metricapproachisasimplemodelwhereweapproachmeasuresfromthreedistinctlevels:
Conceptuallevel(Goal)Operationallevel(Question)Quantitativelevel(Metric)
© 2012 Fiserv, Inc. or its affiliates.
WhatisGQM?
Conceptuallevel(Goal)Agoalisdefinedforateamorsystem,tomeetaspecificpurpose,withafocusondemonstratingameaningfuloutcomeforthebusiness.
© 2012 Fiserv, Inc. or its affiliates.
WhatisGQM?
Operationallevel(Question)Asetofquestionsiscraftedtodefineanobjectivemodelforassessingtheachievementofourgoals.
© 2012 Fiserv, Inc. or its affiliates.
WhatisGQM?
Quantitativelevel(Metric)Asetofmetrics,basedupontheobjectivemodel,isassociatedwitheveryquestioninordertoansweritinameasurableway.
© 2012 Fiserv, Inc. or its affiliates.
• Identifytherightgoalsforateamorsystem• Clarifyourgoalsbyaskingtherightquestions• Identifymeasuresandmetricsthatcananswerthequestions
• Findwaystocollectthedatanecessarytorealizeourmeasuresandmetrics
• Analyzethedatatoassesswhetherweareansweringourquestionsandachievingourgoals
• Usewhatwediscovertoimproveourgoals,questions,andmetrics
HowdoweuseGQM?
© 2012 Fiserv, Inc. or its affiliates.
Example:Mygoalfortoday
Goal
Myaudiencedoesn'tfallasleepinthenexthalf
hour
Question
Didtheyevenshowupinthefirstplace?
Havetheybeenthoroughlycaffeinated?
Aretheypayingattentionsofar?
MetricEmptySeatvsSoldSeat(ActualvsExpected)
SnoretoSilenceRatio(SnoreComplete/SilenceCommitted)
EyeContactVariance
HecklingDensity
© 2012 Fiserv, Inc. or its affiliates.
Redefining the Problem
© 2012 Fiserv, Inc. or its affiliates.
RedefiningtheProblem
InAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…
...ismeetingCommitments
...isProducingQualitySoftware
TheTeam...hasaStableVelocity
© 2012 Fiserv, Inc. or its affiliates.
RedefiningtheProblem
Let’sfigureitout,GQMstyle
InEnterpriseAgile,wefeelprettyconfidentweknowhowtomeasuresuccessbecause…
© 2012 Fiserv, Inc. or its affiliates.
GQMAreastoDiscuss
• DeliveryTeamHealth(Scrum)• TechnicalQuality• ProgramHealth• ProductQuality• PortfolioHealth• PortfolioFinancials
© 2012 Fiserv, Inc. or its affiliates.
GQM at the Delivery Tier
© 2012 Fiserv, Inc. or its affiliates.
Teamscanplan,coordinate,anddeliverpredictablyenoughtomeetarelease
levelcommitment
Hastheteamestablishedastablevelocity?
Doestheteamhavetheresourcesitneedstobesuccessful?
Doestheteamdeliverthecommittedfunctionalityeachsprint?
StoryPointCompletion%(SPsComplete/SPsCommitted)
VelocityVariance
TeamStabilityIndex
Isanythingpreventingtheteamfromdeliveringoncommitments? Blockers
Doestheteamdeliverthroughoutthesprintoratthe“lastminute”?
SprintCompletionRatio(%SPsAcceptedbySprintEnd- 2)
DeliveryTeamHealth(Scrum)Goal MetricQuestion
UserStoryCompletion%(USsComplete/USsCommitted)
Towhatdegreeistheteamencapsulated(i.e.make/meetcommitmentsonitsown)?
%UserStoriesw/outsidedependencies
© 2012 Fiserv, Inc. or its affiliates.
Teamscanfrequentlydeliverworking,tested
softwareofahighlevelofquality
Istheteamabletomaintaintheoveralltechnicalhealthoftheproduct?
Canthequalityoftheproductbeingshippedbeverified? OpenDefects
TechnicalDebt
Aretheteamsabletoaddresstechnicalproblemsinatimelymanner? DefectAging
Aretechnicalproblemsgettingpastourqualitychecks? EscapedDefects
Aretechnicalproblemsbeingdiscoveredinproduction? LatentDefects
Goal Question Metric
TechnicalQuality
© 2012 Fiserv, Inc. or its affiliates.
GQM at the Program Tier
© 2012 Fiserv, Inc. or its affiliates.
ProgramHealth
ProgramManagementcanmakedeliverycommitmentsandsuccessfullymanage
incomingrequests
IstheProgrameffectivelymanagingorchestration/mitigatingdependencies?
Istheprogramdeliveringcommittedfunctionalityeachrelease?
Istheroadmapoffeaturessufficienttoeffectivelymaintainareadybacklog? FeatureRoadmapVisibility
%ReadyBacklogUserStoriesw/opendependencies
FeatureCompletionRatio
HowlongdoesittakeforanOpportunitytomovefromIdeatoDelivery? FeatureLeadTime
Goal MetricQuestion
IstheReadyBacklogsufficientlygroomedtobepredictablyconsumedbytheDT? %ReadyBacklog
IstheProgram(POTeam)focusedondeliveringvalue? %ValueinSprint
© 2012 Fiserv, Inc. or its affiliates.
Goal Question Metric
Theorganizationdeliversanintuitiveexperienceand
productfeaturesthatdelightourcustomersandproducethehighestbusinessvalue
possible
Didthedeliveredsolutionmeetthebusinessneed?
Product/MarketFit(ActualvsExpected)
DoestheOverallProductDelighttheCustomer? NetPromoterScore
DidtheFeatureDelighttheCustomer?PirateMetrics
(Acquisition,Activation,Retention,Revenue,Referral)
ProductQuality
© 2012 Fiserv, Inc. or its affiliates.
GQM at the Portfolio Tier
© 2012 Fiserv, Inc. or its affiliates.
PortfolioHealth
32
Theorganizationdeliversproductfeaturesand
capabilitieswiththebestabilitytodeliverearlyROI
Howlongdoesittakeforanopportunitytomovefrom“goodidea”to”done”?
Isthesystemdeliveringconsistently?
EpicLeadTime
DueDatePerformance
Areweinvestingintherighttypeofwork? PortfolioInvestmentMix
Arewefocusingonlyonthemostimportantthings? ValueStreamWIP
AbandonedWorkArewestartingontoomanythings?
Goal Question Metric
© 2012 Fiserv, Inc. or its affiliates.
PortfolioFinancials
33
Goal Metric
The Portfolio is meeting its strategic, financial
obligations and objectives
Are we getting the right return on our investments?
Are we spending too much just to keep the lights on?
Value to Cost Ratio
CAPEX to OPEX Ratio
Are we getting the expected results from this Portfolio?
Variance to Results($ or % Market Share)
Question
Are we meeting our targeted spend rates? Variance to Spend Target
Are we hitting our targeted value and maintenance blend?
Variance to Value/Maintenance Target
($ or % Spend)
© 2012 Fiserv, Inc. or its affiliates.
Redefining the Problem (Solved)
© 2012 Fiserv, Inc. or its affiliates.
AttheEnterpriseLevel,wefeelconfidentweknowhowtomeasuresuccessbecause…ThePortfolio...consistentlydeliversproductfeatures&capabilitieswithafocusontimetomarketandearlyreturnoninvestment(PortfolioHealth)...consistentlymeetsitsstrategicfinancialobjects(PortfolioFinancials)TheProgram...canmakedeliverycommitmentsandsuccessfullymanageincomingrequests(ProgramHealth)...canensuretheorganizationdeliversthemosteffectiveexperienceandproductfeaturestodelightourcustomersandproducethehighestbusinessvaluepossible(ProductQuality)TheDeliveryTeams...canplan,coordinate,anddeliverpredictably,inordertomeetareleasecommitments(TeamHealth)...canfrequentlydeliverworkingtestedcodeofahighlevelofquality(TechnicalQuality)
© 2012 Fiserv, Inc. or its affiliates.
So now what?
© 2012 Fiserv, Inc. or its affiliates.
üIdentifytherightgoalsforateamorsystemüClarifyourgoalsbyaskingtherightquestionsüIdentifymeasuresandmetricsthatcananswerthequestions
• Findwaystocollectthedatanecessarytorealizeourmetricsandmeasures
• Analyzethedatatoassesswhetherweareansweringourquestionsandachievingourgoals
• Usewhatwediscovertoimproveourgoals,questions,andmetrics
NextSteps
© 2012 Fiserv, Inc. or its affiliates.
Final Thoughts