does15 - jody mulkey - devops in the enterprise: a transformation journey

Download DOES15 - Jody Mulkey - DevOps in the Enterprise: A Transformation Journey

Post on 15-Jan-2017




0 download

Embed Size (px)


PowerPoint Presentation

Ticketmasters Transformation JourneyDevOps In the Enterprise @jodymulkey

Powering Lifes Experiences

Amazon Confidential

Amazon Confidential


Even the longest journey must begin where you stand.-Lao-tzu

Background - Live NationLargest Live Entertainment company in the world.>450 Million fans in 40 countries>120 Venues Owned and Operated>24K Concerts

Every 22 minutes somewhere around the world we are connecting fans with unforgettable moments of joy6

Background - Ticketmaster1976- Founded at Arizona State University1996- launched2010- Live Nation and Ticketmaster join forces2011- Transformation journey begins


230% increase12% increase


Uh oh!


What now?DevOpsto theRescue


Just like Football

Ops = DefenseDev = OffenseX


Just like Football

Ops = Offensive LineDev = Skill positions


DevOps StrategiesEmpathyEmpowermentMetrics

Empathy75 in 75Breathing Customer OxygenEveryone is a fanBreaking bread together

Culture culture culture(Pods)= two pizza teams


EmpowermentTeams crave responsibilityOne team, one mission, one goalDevelopment Teams OncallSelf Service.*

65/75 Scrum teams pushing their own code through the delivery pipeline

Delivery PipelineMonitoring/AlertingLog Aggregation15

MetricsBusiness metrics > System MetricsOutcomes > OutputsInstrument everythingDemocratize the data

Protip make it easy and scalable


Lessons LearnedChange is hard

Lessons LearnedEmpowered Expertise = Lower MTTR

Abacus story18

Support at the EdgeFormalized tiered support modelMove support as close to customers as possibleFocuses teams on projects that prevent problems

ProjectsTOC transformationNOC->TOC transitionEmbedding of teams19

Clients and Fans

ProjectsTOC transformationNOC->TOC transitionEmbedding of teams20

Support at the Edge - TOC

Technical Operation Center(TOC)=EMT

EMT, they have 15 minutes to get the incident resolved on the way to the hospital21

Support at the Edge - SRESite Reliability Engineers(SRE)=Emergency Room Doctors

They are experts in incidents & trauma, the know how all of the systems of the body work and how they are connected. They have 30min to resolve the issue22

Support at the Edge Production EngineeringProduction Engineers=Surgeons

Specialists have intimate knowledge of the systems that they are working on23

Before Support at the EdgeAverage MTTR 47 Minutes

After Support at the EdgeAverage MTTR 3.8 Minutes

Lessons LearnedDevOps has nothing to do with technology

Abacus story26

Ticketing Engine (aka The Host)Powers $25B in commercePerformance measured in microsecondsCode first committed 1976VMS running on emulated VAX

Bedrock TeamDeploys their own codeIntegrated PagerDuty for Oncall AlertsLeverages Cucumber/BDD for automated functional testsTicketing Engine (aka The Host)

Lessons LearnedThere are only two states:You own the systemorThe system owns you

Abacus story29 (AKA TMOL)Drives 40% of revenueMajority of system built in 2000 ModPerlTribal knowledge no longer at the company


Metal to Money No HandsDevelopment team taking over svc delivery Push Button Deployment of the TMOL stackTribal knowledge encapsulated in code

Insert picture from team32

Closing Thoughts Ego is a force field for learning

To know that you do not know is the bestTo pretend you know when you do not know is a disease-Lao-tzu