does16 san francisco - scott prugh & erica morrison - when ops swallows dev

CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.

When Ops Swallows DevOur Cultural and Technical Journey to Realize DevOps at CSG

Scott Prugh, Chief Architect & VP Software Development & Operations, [email protected], @ScottPrughErica Morrison, Director Software Development, [email protected], @ericarmorrison

mailto:[email protected]



When Ops Swallows Dev

Techniques(v2)1. Holistically Improve Work Visibility

a) Incidentsb) Dependenciesc) Single Intake of Planned Work

2. Challenging Shared KPIs3. Go See & Role Rotation4. Infra as Code & Shared Understanding5. Legacy Test Automation / ATDD6. Telemetry & Shared Understanding

Pressures & ConstraintsDemand for Quality & Speed

SoRs being pressured to become SoEsOrg & Process Debt, Technical Debt

Strive for Unimodal ITInvest in:

Culture, Empathy, Understanding,Simplicity & Automation

Techniques(v1)1.Accelerate Learning & Lean

Thinking2.Inverse Taylor Maneuver3.Inverse Conway Maneuver4.Shared Service Continuous Delivery5.Environment Congruency &

Practice6.Application Telemetry7.Visualize Your Work8.Work Release & WIP Limits9.Cadence & Synchronization10.Reduce Batch Size

Accelerate Feedback & LearningUnderstandingAccountability

Engineering

2014 2015 2016

Journey and Principles1. Org Archetype Journey 2012-2016

Creating Service Delivery Teams2. What we learned….3. What is next…

Scott Prugh, Chief Architect & VP Software Development & Operations, [email protected], @ScottPrughErica Morrison, Director Software Development, [email protected], @ericarmorrison

Our Cultural and Technical Journey to Realize DevOps at CSG

YouTube: f4et0EGvKXA YouTube: tKdIHCL0DUg



https://www.youtube.com/watch?v=f4et0EGvKXA

https://www.youtube.com/watch?v=tKdIHCL0DUg

CSG International Confidential and Proprietary Information Copyright © 2015 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved. CSG International Confidential and Proprietary Information Copyright © 2016 CSG Systems International, Inc. and/or its affiliates (“CSG International”). All rights reserved.

CSG in NA: Who Are We?SaaS Based Customer Care and Billing

• ~50M Subscribers • 100k Call Center Seats• ~6B External Transactions/month• 40 Dev. Teams & 1000 Practitioners• ACP: ~20 Technology Stacks: JS to HLASM

• Integrated Suite of 50+ applications

Challenges:• Time to Market & Quality/Release Impact• Technology Stovepipes• Role Stovepipes• Operational Quality

Print and Mail Factory• ~70M statements/month• Lean & Efficient

Challenges• Continuous Optimization

http://upload.wikimedia.org/wikipedia/en/b/b5/Eastlink_logo.svg


Context: Opposing Forces

Development Operations

Change ManagementRelease ManagementProduction Operations

PMO

Speed! Stability!Your code sucks!Your environments suck!

Your SR tool sucks! Your Agile tool sucks!We hate CRQs!We hate, CRQs!

Customers:

High Quality Features Quickly!

© Alphaspirit @ 123RF.com , Wavebreakmediamicro @ 123RF.com, Warrengoldswain @ 123RF.com


Courage and FeedbackAutomated Unit TestsAutomated System TestsCI Build w/ Automated TestsExploration TestsManual regression testingOperational Quality/FeedbackAnalysis of production metrics(Telemetry)Customer Opinion

Time

DOES15 - Elisabeth Hendrickson - Its All About Feedback https://www.youtube.com/watch?v=r2BFTXBundQ

Latency: How do we move from days to hours?

Faster Loops means Faster Learning


Functional Archetype: Role specific organizations “locally optimize” around their structure and role instead of globally optimizing the flow of business value.

Customer

Reqs. Design Dev. Testing Ops.

Production

Opti

mize

Opti

mize

Opti

mize

Opti

mize

Opti

mize

Flow of Business ValueBig product/feature a client wants

Queues

CSG Pre-2012

Role ShapeQueuesWaitTime=(%Busy)/(%Idle)Queues delay feedback and learningQueues help enforce I-Shaped resources

http://culturecartography.files.wordpress.com/2012/07/picture-171.png


Agile Archetype: Cross functional, self-empowered agile teams optimize the SDLC value-stream and remove impediments to do so.

Customer

Ops.

Production

Opti

mize

Flow of Business ValueBig product/feature a client wants

Agile DBT Team(7+/-2)Agile DBT Team(7+/-2)

Agile DBT Team(7+/-2)CSG 2012-2015

Role Shape

Batch Size Reduction

Test AutomationCadence & Synchronization

Shared Operations Teams

Inverse Conway & Taylor

Work Visibility Continuous Delivery V1

“little t”


Release Incidents ImpactScore I-ImpactScore13.1(04-07) 201 4550%(1x)14.1(01-12) 76 15666%(3x)14.2(04-13) 38 8681%(5x)14.3(07-27) 39 8382%(5x)14.4(10-26) 47 11176%(4x)15.1(02-08) 20 4091%(11x)15.2(05-17) 33 8182%(6x)15.3(08-09) 18 4391%(11x)15.4(11-15) 26 6885%(7x)16.1(03-06) 36 8282%(6x)16.2(06-12) 28 7085%(7x)16.3(09-25) 24 5089%(9x)

Results: Release Quality Improvements

CSG 2014-2016

~10x quality

50% TTM


System View of Quality

Facts1. 98% of incidents outside release2. 92% of incidents fixed by ops

OperationsDevelopment

Your code sucks!Works in test!

Incidents Resolved By Group/Priority 12mo period ending 3/16

My code does suck


Before: Traditional Dev and Ops Structure

Hypotheses• Different org goals were working against system goals• Lack of op understanding = hard to run software• Lack of shared mission = lack of empathy across teams• Handoffs caused elongated lead time• Lack of engineering skills in operations prevented

improvements and encouraged duct-tape engineering

Observations• Release quality optimized but system quality not• Manual processes around hard to run software• Dev lacks understanding around operations

• Lack of feedback into development• Collaboration was unnatural and scripted

• Lack of “Esprit De Corps”• Lots of context switching and work management chaos

Market Archetype*: Service Delivery(DevOps) Teams


Before: Traditional Dev and Ops Structure After: Cross Functional DevOps Structure

The future is here*IAC vs IAT

CSG 2016+

(March

2016)

Market Archetype*: Service Delivery(DevOps) Teams


Market Archetype*: Team Level

Architect

Product Owner

Prod Ops

Developer

QA

SM

Platformvia Self

Serve(TBD)

Service Delivery Teams Build and Run their own software. This shortens feedback, increases learning and rapidly improves quality.

Why Bring Dev and Ops Together?Understanding: Team now has the ability to understand, learn and improve the delivery chain.

Accountability: Teams now accountable for the majority of the delivery chain: Design, Build, Test, Operations.

Engineering: 1) Team can inject and cross train engineering principles/skills into operations, 2) Team can now evolve operations from being solely a process activity to an engineering activity with true continuous improvement. Shift left: no duct-tape engineering.

Other Benefits: Communication, Meetings, Planning, Collaboration, Work Visibility, Shared Leadership Vision

CSG 2016+

(March

2016)

Role Shape

“Big T”

http://culturecartography.files.wordpress.com/2012/07/picture-131.png


What We Learned


Going All In


Title: “When a Dev Manager Takes Over

NLB -> The Horrors and Realizations that

Follow”

Title: "Pizza, Beer, and Illegal Drugs - An

Overview of Motivational

Techniques in DevOps"

Timing: “We should present something on

NLB, but we would need to present at

1AM”

A Whole New World


I’m In the Phoenix Project!

Invisible work/Work in multiple systems

Competing priorities from across the company

WIP/Overutilization

Manual configuration

Technical debt

Brent

Poor visibility into our specific changes


Applying DevOps

• Made resource changes

• Implemented automated reporting

• Applied dev best practices

• Implementing configuration as code

• Completed work tracking system overhaul

• Changed workload management

• Increased change process visibility


Ops is hard!

Change process and volumes can be daunting

Change can be scary

Application architecture is needed

Enablement is key

Support is truly 24x7

Ops is forced to tolerate a lot of pain

Ops from a Development Perspective


Summary / What’s NextAccelerate Feedback & Learning:

Understanding, Accountability, Engineering

Challenge Org Norms(Courage)Structure for Feedback/Outcome Speed, Quality

Things LearnedWork Chaos: Change and BAU

Manual ProcessesResource Bottlenecks

Tech DebtTesting(lack of)

ProcessChange: Lead Time, SDLC Rigor, Ownership

Bridge ITIL/SDLC(Process & Tools)Impact Reduction

Centers of Enablement(vs Excellence)

TechnologyMore engineering less duct-tape

Mainframe ASM Java: 30-1 reductionCD v2.0: From IAT to IAC/Cloud Pipeline

PeopleEngineering Culture

Cross Skilling / Upskilling

What’s Next….


Change Lead Time & Schedule Buffers

2.5 Scheduling Policy:

Minimal lead time for a change classified as

“normal” is 5 calendar days. Exceptions may

occur as to meet customer or business needs.

Make Change

Great Again!