does15 - jody mulkey - devops in the enterprise: a transformation journey

Download DOES15 - Jody Mulkey - DevOps in the Enterprise: A Transformation Journey

Post on 15-Jan-2017

816 views

Category:

Business

0 download

Embed Size (px)

TRANSCRIPT

PowerPoint Presentation

Ticketmasters Transformation JourneyDevOps In the Enterprise @jodymulkey

Powering Lifes Experiences

Amazon Confidential

Amazon Confidential

Videohttps://www.youtube.com/watch?v=TIA3N52vm70

Even the longest journey must begin where you stand.-Lao-tzu

Background - Live NationLargest Live Entertainment company in the world.>450 Million fans in 40 countries>120 Venues Owned and Operated>24K Concerts

Every 22 minutes somewhere around the world we are connecting fans with unforgettable moments of joy6

Background - Ticketmaster1976- Founded at Arizona State University1996- Ticketmaster.com launched2010- Live Nation and Ticketmaster join forces2011- Transformation journey begins

7

230% increase12% increase

8

Uh oh!

9

What now?DevOpsto theRescue

10

Just like Football

Ops = DefenseDev = OffenseX

11

Just like Football

Ops = Offensive LineDev = Skill positions

12

DevOps StrategiesEmpathyEmpowermentMetrics

Empathy75 in 75Breathing Customer OxygenEveryone is a fanBreaking bread together

Culture culture culture(Pods)= two pizza teams

14

EmpowermentTeams crave responsibilityOne team, one mission, one goalDevelopment Teams OncallSelf Service.*

65/75 Scrum teams pushing their own code through the delivery pipeline

Delivery PipelineMonitoring/AlertingLog Aggregation15

MetricsBusiness metrics > System MetricsOutcomes > OutputsInstrument everythingDemocratize the data

Protip make it easy and scalable

16

Lessons LearnedChange is hard

Lessons LearnedEmpowered Expertise = Lower MTTR

Abacus story18

Support at the EdgeFormalized tiered support modelMove support as close to customers as possibleFocuses teams on projects that prevent problems

ProjectsTOC transformationNOC->TOC transitionEmbedding of teams19

Clients and Fans

ProjectsTOC transformationNOC->TOC transitionEmbedding of teams20

Support at the Edge - TOC

Technical Operation Center(TOC)=EMT

EMT, they have 15 minutes to get the incident resolved on the way to the hospital21

Support at the Edge - SRESite Reliability Engineers(SRE)=Emergency Room Doctors

They are experts in incidents & trauma, the know how all of the systems of the body work and how they are connected. They have 30min to resolve the issue22

Support at the Edge Production EngineeringProduction Engineers=Surgeons

Specialists have intimate knowledge of the systems that they are working on23

Before Support at the EdgeAverage MTTR 47 Minutes

After Support at the EdgeAverage MTTR 3.8 Minutes

Lessons LearnedDevOps has nothing to do with technology

Abacus story26

Ticketing Engine (aka The Host)Powers $25B in commercePerformance measured in microsecondsCode first committed 1976VMS running on emulated VAX

Bedrock TeamDeploys their own codeIntegrated PagerDuty for Oncall AlertsLeverages Cucumber/BDD for automated functional testsTicketing Engine (aka The Host)

Lessons LearnedThere are only two states:You own the systemorThe system owns you

Abacus story29

Ticketmaster.com (AKA TMOL)Drives 40% of revenueMajority of system built in 2000 ModPerlTribal knowledge no longer at the company

30

Metal to Money No HandsDevelopment team taking over svc delivery Push Button Deployment of the TMOL stackTribal knowledge encapsulated in code

Insert picture from team32

Closing Thoughts Ego is a force field for learning

To know that you do not know is the bestTo pretend you know when you do not know is a disease-Lao-tzu

34