why don'we just say "no" - devopsdays silicon valley jun 2014

Why don’t We just say no? !

!

!

Ge$ng!things!done!doesn't!always!

start!with!"Yes". @dominicad!www.ddegrandis.com!

@dominicad!www.ddegrandis.com!

For a number of reasons, it’s hard to say no.!

@dominicad!www.ddegrandis.com!

#1.!!We!do!stuff!for!people!we!like.!!!

!

@dominicad!www.ddegrandis.com!

Ask!your!neighbor!this!quesCon….!

!!

“Why!do!you!take!on!more!work!than!

you!have!capacity!to!do?”!!!

@dominicad!www.ddegrandis.com!

Reason!#!2!–!We!are!team!players!

@dominicad!www.ddegrandis.com!

Reason!#!3!–!Fear!from!those!in!posiCon!of!power!

@dominicad!www.ddegrandis.com!

Reason!#!4!–!People!pleaser!

@dominicad!www.ddegrandis.com!

Why do Development teams seem to be able to say no

more than Ops teams? !

Stuff Ops deals with that Dev traditionally doesn’t: •  Live prod issues •  DDOS attacks •  100% disk utilization •  access requests at 2:00 am •  opening up firewalls

Over the last 11 years, a lot of Dev teams have been given permission to say,

“We’ve already done the planning for

this week. Your next opportunity to get

your request considered is next week.”

Saying “No” has been institutionalized by using timeframes.

This doesn’t work for Ops very well… “Sorry - that live production fix will have to wait until next week” What if instead of using timeframes to

say “No”, we used limits to say “No”?

A!look!at!one!team’s!evoluCon!

40 Ops Engineers (SysAdmin, DBA, Network, Mon, Sec)

Tasked to build out/retrofit 6 data-centers across 6 diff countries.

AND…. •  keep the lights on four existing data centers •  build out a new platform architecture •  support live issues (on-call) •  roll out a new configuration management tool •  deploy a crap load of new features •  deal with 3 reorgs over a 6 month period !!

!

!

!!Problems:

•  Conflicting priorities •  risky dependencies •  over-commitments •  interrupt-driven context switching

resulting in missed commitments. •  Confusion around the new org structure !!

!

!

How!we!got!started!on!road!to!

improving!–!looked!at!3!data!pts!

!!

!

open closed

#1 Can we keep up with the demand?

0!

10!

20!

30!

40!

50!

60!

1!

2!

3!

4!

5!

6!

7!

8!

9!

10!

20!

30!

40!

50!

60!

70!

80!

90!

100!

110!

120!

130!

140!

150!

160!

170!

180!

190!

200!

250!

300!

More!

#"#ckets"

# days it took for ticket to go from created to closed

#2 Lead time – how long does it take to get work done?

Validate Design Deliver Done Implement

doing

Maint"

Projects

Implement

done !!

96

#3!Where!is!work!stuck?!!

Customer!Mtg!Cme!

!!

Invited!customers!to!a!meeCng!and!showed!them!the!data.!!–  showed!them!the!demand!and!what!was!ge$ng!done!!!

–  showed!them!the!Lead!Cme!!

–  showed!!where!work!got!stuck!!!

Customers!appreciated!the!visibility!into!Ops!

!We!took!advantage!of!that!by!humbly!asking!for!their!help.!

!beginning!with!all!of!those!Cckets!si$ng!in!the!validate!state.!!

Next!steps!

The!introducCon!of!a!work`in`progress!(wip)!

limit.!!Some!of!these!guys!had!20!–!40!Cckets!in!

their!queue.!!We!asked!them,!“Does!this!seem!

reasonable?”!

!

How!about!10?!!!

Let’s!head!in!that!direcCon!and!see!what!

happens.!!!

!

and!then!there!were!reorgs!

The!1st!!org!restructure!change!created!the!A!Team!!`!to!focus!on!compleCng!projects!close!to!being!done,!but!sCll!hanging!on.!

!

•  This!team!didn’t!have!to!respond!to!one!off!requests!and!wasn’t!supposed!to!be!on`call.!

!!

•  2nd!org!structure!change!split!Ops!into!3!teams!(live!Ops,!Build,!architecture)!

•  Live`Ops!with!25%!of!team!and!60%!of!the!work!!

!

!

•  access requests for systems, non-Zabbix monitor •  hardware investigation/verification/fixes -  vlan/port changes -  data retrieval (i.e. logs, network stats, etc) •  configuration triage - firewalls, load balancers,

OS settings •  (small) capacity expansion •  verification of configs/services across shards •  database development consultation •  security compliance mitigation

Live Ops tasks

Live!Ops!SRE!Changes!

  •  Socialized wip limit idea over 6 months and

gradually lowered it from 10 to 7 – out of 18 guys, average is 5-7.

•  Hired 4 more people, although 2 got stolen by another team

•  Closed out all tickets with no activity > 90 days

•  Started saying “No” to last minute requests. !

Hi D.C., Team SRE has a very large number of changes scheduled for today already, and an even larger number of requests in our backlog that this request will displace if moved to the front of the queue. It would not be fair to other teams if we jumped on this immediately while planned work is pushed off. Monitoring should be a requirement for a service to go live, not a last minute addition. For us to fully support a live service, please implement monitoring before going live. For future requests, please give us as much notice as possible, and make sure to create a ticket (xxx.com) so we can prioritize and schedule the changes as necessary. Here's the ticket for this work…. Respectfully, A.H !

Live!Ops!SRE!changes!con’t! 

•  Took time during standups to focus on kaizen improvements.

•  Reduced validate state from 7 to 5 to 3 days.

•  Found creative way to deal with walkups, and work done via personal relationships

•  15 min daily sync up at 3pm instead of interrupting.

•  5 min videos to present Ops review data to customers

!

!

A few simple rules

Clear definitions of done between queues

kaikzen/��!noun, continuous improvement A business philosophy or system based on making positive changes on a regular basis.

“Disney thinks

we can’t do it.

Let’s prove

them wrong.” -Steve Jobs

“Asking this much of people,

even when they wanted to give

it, was not acceptable.”

!

!! ! ! ! !!`!Ed!Catmull!

“WIP it down

to the personal safety level”!

“No” is an honorable reply to

someone asking too much

from you.

“The!takeaway!for!leaders!is!

realizing!the!power!you!have!

over!other!people!when!you!!

ask!something!of!them.”!

!!!

!!!!!!!!!!!!!!!!!!!!!!!!!!`!Vaness!Bohns

!!

!

!

But,!what!about!when!the!show!must!

go!on?!

!

World!Cup!finals!`!!Sunday,!July!13!

!!

fifa regulations ensure they have capacity. “pg 8 e) ensuring the presence of a sufficient number of

ground staff and security stewards to guarantee safety.”

Let’s not expect day shift workers to also cover the night shift.

@dominicad!www.ddegrandis.com!

Honey-do list rules for saying no to the spouse…

@dominicad!www.ddegrandis.com!

Credits!!

hqp://www.psychologytoday.com/blog/caveman`logic/201403/why`so`

many`people`just`cant`say`no!

!

The!Book!of!NO:!250!Ways!to!Say!It!—!and!Mean!It!—!And!Stop!People`

Pleasing!Forever.

hqp://www.amazon.com/The`Book`No`People`pleasing`Forever/dp/

0071460780!

!

Vanessa!Bohns!!`!!social!psychologist!and!professor!of!management!

sciences!at!the!University!of!Waterloo!in!Ontario!

hqp://hereandnow.wbur.org/2014/03/31/saying`no`psychology!

!

eBay!arCcle!!hqp://www.uie.com/arCcles/death_of_relaunch/!

!

Fifa!regulaCons!

http://resources.fifa.com/mm/document/tournament/competition/01/47/38/17/regulationsfwcbrazil2014_update_e_neutral.pdf !