does16 london - better faster cheaper .. how?

Better, Faster, and Cheaper. How?

Damon Edwards John Willis

Co-Founder & Managing Partner DTO Solutions, Inc.

Damon Edwards @damonedwards

Director of Ecosystem Development Docker, Inc.

John Willis @botchagalupe

devopscafe.org

30x 200xmore frequent deployments

faster lead times

60x 168xthe change success rate

faster mean time to recover (MTTR)

2x 50%more likely to exceed profitability, market share & productivity goals

higher market capitalization growth over 3 years*

High performers compared to their peers…

Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

Recent IT Performance Data is Compelling

30x 200xmore frequent deployments

faster lead times

60x 168xthe change success rate

faster mean time to recover (MTTR)

2x 50%more likely to exceed profitability, market share & productivity goals

higher market capitalization growth over 3 years*

High performers compared to their peers…

Data from 2014/2015 State of DevOps Report - https://puppetlabs.com/2015-devops-report

Recent IT Performance Data is Compelling

Faster

HigherQuality

MoreEffective

Fast

CheapGood

“Pick Two!”

Conventional Wisdom

Faster, Better, and Cheaper?

Faster, Better, and Cheaper. How?

Culture Automation Measurement Sharing

Better, Faster, Cheaper. How?

Org structure that stays out of the way


Dev Release OpsPlanning

Application Knowledge

Operational Knowledge

Business Intent

Handoff!

Handoff!

Handoff!

Ownership but limited Accountability

Accountability but no Ownership

Silos are enemy of throughput and stability

Dev Release OpsPlanning

Focus on reducing handoffs and improving flow

Service Aligned Delivery Teams



Environments as a

Service

QA as a

Service

Insights as a

Service

Security as a

Service

Environments Team

QA Team

Monitoring & Metrics

TeamSecurity

Team

Org structure that stays out of the way Build everything through a SDLC


Average Peformers: SDLC for applications

Code

Source Repo

Config

CI Server

Package Repo

Operations Console

Application Deployment Automation

Pre-Production Environments

Production Environment

Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Tests

Application Deployment Automation

Code

Versioned Release

Source Repo

Config Env Spec

Run- book

Auto-mation

CI Server

Package Repo

Operations Console

Pre-Production Environments

Production Environment

Packages

Ope

ratio

ns

Dev

elop

men

t

SOURCE

Dev Ops *

Collaboration

Dev Ops *

Execute Operations Procedures

Same People!!

Tests

High Performers: SDLC for everything

Org structure that stays out of the way Build everything through a SDLC Make the work visible


Service now

Parts Unlimited - "Major Release 6"

Early 2014

Project Initiation

ZRA (finance)

Approve Project

Monthly Steering Meeting

Portfolio

C-level

Steering Comittee

Provides Input

Project Charter

High-Level• Stories• Project Info• Description• Budget• Schedule

PMStakeholders (Tech and Biz)

Create Work Breakdown

Work Breakdown (MS Proj)

High-Level• Milestones• Resource

Planning

3 months 3 monthsHold / Pause

Create Requirements

(Project Meeting)

MS Office

• Detailed Req for new features

• Technology refreshes

• ERD (Infra req)• DRD (Dev req)• BRD (Biz req)

Share Point

Create Design

Tech ReqTech

ReqTech Req

Tech Leads Architects Vendor Arch

Ops Arch

High-LevelServer Tickets

3 months

Receive Request for

Servers

Create Server

Request Spreadsheet

ServerReq

PMTixattach

Route for Approval

Tix

1 week 1 week

• Budget• Appropriate

Resources DB

App or Web

orApproved Into Ops

Delivery Queue

Delivery Manager

"Matt"

Service now

"Heads up"

Assign to Delivery Engineer

Delivery Engineer

Clarify or Confirm Req with Dev or

QA

1 - 6 weeks

Provision Server

and Rework

DBA Validation

App/Web Validation

RestoreData

1 weekApp

Team

App Team

PMStakeholders (Tech and Biz)

Dev Leads

4 weeks

ARB Queue

Detailed Analysis and Requirements

Jira "Stories"

Maybe

Track Ticket Dependencies

Confluence Pages

Team Leads and PMs

Assign Requirements

add more detail for their teams

Architecture Review Board

"Bill" plus Architects

Working Group

Ops? (sometimes)

Devs, PM, Engr, QA

Development Sprint

2 week c/t

Existing Dev Environments

Acquire / Prepare needed

dataOps DBA

Service Data Setup

(Mainframe)

"Jennifer"

Test Data Configuration

Manager

Development Deploy to Integration

Dev, QA

Integration & Regression

Testingfocused on service

ScrumDev/QA

Integ03

ScrumDev/QA

Test Link

Sprint Review

Release to Prod

Product Owners(Using own

criteria)

Create CAB ticket

or

Scrum Team Ops Team(if legacy)

Push Deployment to Stage

Stage

Email Notification

Jira

NewArch

Build VMs

Jira

Ops

ServiceNow

Legacy

QA LeadPMsQAs

End to end testing in Prod

Prod Env

PrdDB

Go-No Go decision meeting

Team Leads

Jira

Ops

By Cluster

"Remove Feature Flag"

(if new arch)

16 weeks

6 weeks H/C: 6 3 weeks H/C: 8

4 weeks H/C:8 3 weeks H/C: 14

Data Setup Integration Testing

DEv Arch

Create Change Tickets > 100

Service Now

ComputeNet

FacilityCablingStorage

"Linda"Ops PM

RESET DELIVERY

DATE!

Steering Comittee

Fix Tickets!

"Linda" Ops PM

Dev Leadership

Assign Dev Team

Ops Intake Meeting

Dev Leadership

1 week

GroupCIOs and

Arch Leads

QA

SteeringDesign

Dev BreakdownDev / Test

Staging Release

Server Requirements GatheringServer Approval and Assignment

Provisioning

Production Release

Initiation and Planning

Create OpsTickets

TS PD

TS PD

Gaps in Requirements• Licenses• Dependencies on 3rd party apps• Capacity planning always seem low

("robbing Peter to pay Paul")• Don't purchase in advance even though

we know it's coming

Duplicate info across different documents

EP

D

D

Procurement of physical servers can take months (lead times for procurement plus facilities groups)

Too many Env. in on ticket cases audit confusionPiecemeal requests ("2 this week, 3 next week")

1 queue for delivery team with ~1,000 tickets at once

Capacity issues cause delay

Often told to stop everything and do something else

TS

D

M

TS

M W

W TS EP

HNo monitoring or backup for some environments

30% of delivery teams time spent "consulting" on performance and dealing with unfounded requests for more capacity

3-5 days to fix~10% S/R

H

D M

TS

H

Often skips CAB. What CAB reviews is often not what built

All manual setup. 1 person really knows how. Low data quality.

Manual process with lots of back and forth.

Many tickets with mismatched priorities

Mostly manual testing

Manual, per clusterFrequently down.

External service updates take offline. Lots of contention.

EPM

D

PDM W

TS

TS D

M TS

PDM

M

S/R - 90%

S/R - 55%

S/R - 15%

D

S/R - 20%

S/R - 50%

Sometimes submits server requests directly to delivery Ad-hoc requests get

lost, maybe 2-3 week delays

TS

High Level

S/R - 75%

9+ months of planning before implementation starts

(and information / requirements still incorrect or incomplete!)

Dev and QA told to submit sever request 6-8 weeks in advance (only done 50% of time)

W5. New "white glove" engagement model

3. Standard product catalog("Environments on Demand")

2. Visualization of flow of work and expected upcoming work

4. Shorten from Design to Implementation

1. Fully Automated Environment Provisioning

7. Small Batches

8. Write end-to-end customer

func. tests

11. Resolve interface to

legacy

10. Test data setup

automation

13. Dev Deploy to Prod for legacy

14. Unify change

management tools

15. Tool

9. Service Verification test writing: shift left to Dev(test early)

12. Remove Bottleneck and Environment Contention(test more)

• Make the work visibile for all • Build alignment and consensus across team

boundaries • Empower teams to find and fix what is getting in the

way

Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery


• Variety• Determine your variety of offerings

based on operational efficiency and market demand

• Velocity• Maintain a steady flow through all

processes of the supply chain• Variability

• Manage inconsistencies carefully to reduce cost and improve quality

• Visibility• Ensure the transparency of all

processes to enable continuous learning and improvement

Immutable Delivery helps realize Toyota’s 4VL

Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices


Microservices

“Small Autonomous Services That Work Together”

-Sam Newman

“Loosely Coupled Service Oriented Architectures With

Bounded Context” -Adrian Cockroft

Working with Monlithic Services

Slide stolen from Adrian Cockcroft

Release Plan

Developer

Developer

Developer

Developer

Developer

QA Release Integration

Ops Replace Old With New

Release

Monolithic service updates

Works well with a small number of developers and a single language like php, java or ruby

Working with Monlithic Services


Release Plan

Developer

Developer

Developer

Developer

Developer

QA Release Integration

Ops Replace Old With New

Release

Bugs

Bugs

Monolithic service updates

Works well with a small number of developers and a single language like php, java or ruby

Working with Microservices


Developer

Developer

Developer

Developer

Developer

Old Release Still Running

Release Plan

Release Plan

Release Plan

Release Plan

Deploy Feature to Production




Bugs


Immutable microservice deployment scales, is faster with large teams and diverse platform components

Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices Respect for People


High-Performers care for their people like strategic assets

• Diversity • Empathy • Areas of Worklife Scale (AWS)

It’s not just humane, it’s a business advantage

Recipe for burnout is inverse of recipe for success

Work overload: Job demands exceed human limits.

Lack of control: Inability to influence decisions that affect your job.

Insufficient rewards: Insufficient financial, institutional or social rewards.

Breakdown of community: Unsupportive workplace environment.

Absence of fairness: Lack of fairness in decision-making processes.

Value conflicts: Mismatch in organizational values and the individual’s values.

Source: Dr. Christina Maslach’s six organizational risk factors that predict burnout (via Dr. Nicole Forsgren)

Org structure that stays out of the way Build everything through a SDLC Make the work visible Immutable Infrastructure Delivery Microservices Respect for People


Co-Founder & Managing Partner DTO Solutions, Inc.

Damon Edwards @damonedwards

Director of Ecosystem Development Docker, Inc.

John Willis @botchagalupe

devopscafe.org