puppetconf 2016: successful puppet implementation in large organizations – james sweeny, puppet

Successful Puppet Implementation in Large Organizations James Sweeny Puppet, Inc.

You will hear me blather about:

DevOps and Puppet

Large vs. Small/Medium Orgs

Deployment Strategies

Common Roadblocks and Pitfalls

Who am I?

This talk is not about…

●  Scaling Puppet

●  High Availability

●  Monitoring

●  Security

Section ## (if desired) and/or Subtitle

Leading platform. Datacenter standard.

Experience Founded in 2005

Scale More than 10 million nodes managed

Ecosystem Deep partnerships with datacenter titans

Customers 1,000+ enterprise customers

Community 4,200+ community-contributed modules

Users 30,000+ organizations using Puppet

Backing

DevOps - CAMS

●  Culture

●  Automation

●  Measurement

●  Sharing

How most companies see it

●  Culture

● Automation ●  Measurement

●  Sharing

How they should see it

● Culture ●  Automation

●  Measurement

●  Sharing

Puppet is (just) a tool.

Puppet doesn’t…

●  Do everything

●  Fix broken culture

●  Shuffle the org chart

●  Train your staff

●  Get you out of an abusive MSP relationship

But it can… ●  Help you to understand your own

systems

●  Provide a focal point and a Lingua Franca

●  Save time and open new opportunities

●  Act as a force multiplier for velocity, consistency, and reliability

Puppet code is state!

Infrastructure as Code ●  Puppet code describes the entire system

(or everything you care about)

●  The code is the system

●  Changing the code is the same as changing the server

●  Designing, provisioning, changing all rolled into one

What is a “Large” Organization?

Large Organizations…

●  Complex org chart

●  Separation of concerns

●  Many teams

●  Many distinct business functions

●  Headcount, not node count

Naïve Deployment Example

Server

Application

In house software, provides business value

Server

Middleware Apache, IIS, Oracle, etc.

Server

Management Netbackup, SEP, Nagios agents, etc.

Server

OS Windows, Linux, Solaris, etc.

Server in a small company

OS

Management

Middleware

Application

Developer(s) Operations/IT

Developer(s) Operations/IT

DevOps!

OS

Management

Middleware

Application

Developer(s)

Operations/IT

Problem in Large Organizations

●  Many “development” teams

●  “Operations/IT” is highly fractured

●  Many overlay teams

App teams

Release team

Middleware

Windows

Security

Linux

… But it’s usually worse ●  Multiple variants of each team

●  Windows/Linux/Solaris/AIX…

●  Secure vs. Non-secure

●  Different team for each Management product

●  Overlays for compliance, security, governance…

●  Changing ownership depending on lifecycle stages

Crazy Org Charts ●  CIO

●  SVP Engineering

—  Middleware

—  Windows Engineering

—  Unix Engineering

—  Backup Team

—  Monitoring Team

●  SVP Application Engineering

—  Hundreds of app teams

—  Release Engineering

●  SVP Security and Compliance

●  …

Journey, not a sprint

So what *does* work? (Reduce the problem domain)

Start small, shallow, and wide Option #1

Shallow and Wide

OS

Management

Middleware

Application

OS

Management

Middleware

Application

OS

Management

Middleware

Application

Single Team

Unmanaged

Shallow and Wide

●  Start with one existing team (usually the one with root/Administrator)

●  Do minimal work

●  Get comfortable with Puppet and version control

●  Don’t boil the ocean

Shallow and Wide (evolution)

●  Add new teams as responsibilities expand

●  Share code repository

●  Central module repository

●  Slowly work up the stack

Platform as a Service Option #2

Platform as a Service

OS

Management

Middleware

Application

OS

Management

Middleware

Application

OS

Management

Middleware

Application

Single Team Unmanaged (Legacy)

Platform as a Service

●  Puppet is hidden from customers (App/Dev teams)

●  Limited, but complete stacks provided

●  Likely need to build/use an additional interface

●  Exceptions to normal policies

Platform as a Service (evolution)

●  Accept Puppet code contributions from other teams

●  Add additional Middleware and OS platforms

●  Decommission/migrate legacy servers

Which is right for us?

Shallow and Wide

●  Most successful real-world

●  Most flexible early on

●  Easiest way to learn your org

●  Gives time to evolve

●  Business value not as easily apparent

PaaS

●  Single team can own and slowly grow

●  Very high upfront cost

●  Quick wins for new applications

●  Side-by-side with legacy for longer (or forever)

●  No benefit to legacy fleet

Other Options

Hybrid Approaches

●  Grant limited Hiera exposure

●  Mix of Options 1 & 2

●  Puppet as a Service

●  Multiple agents

The “Puppet Team”

“Puppet Team”

●  Over time, role becomes advisory

●  Own the initial workflow, Puppet infrastructure, supporting services

●  Become Puppet SMEs

Who is the “Puppet Team”

●  Technical experts on your existing systems

●  Can cut red tape

●  Have root/Admin already

●  Eager to learn, excited to solve problems

●  Dedicated to big picture success

Likely Roadblocks

“That’s not how we do it” The internal “standard”

Battling “That’s not how we do it”

●  Prefer Industry standards to esoteric ones

●  Software owners must articulate reasoning

●  Make your case to “SMEs”

●  Use political pressure as a last resort

Plan, Build, Run Probably the reason everything is awful

Plan, Build, Run

●  Diffusion of responsibility

●  Conflicting sources of truth

●  Destroys feedback loops

●  Massively slows change

●  Puppet makes it obsolete

Politics

Politics

●  Have well connected people committed to Puppet

●  Need strong management

●  Understand other groups needs

●  Be willing to prove real value before winning other groups

●  “Puppet won’t steal your job”

Final Advice

More tips

●  Learn/know your own systems

●  Always question internal dogma

●  Limit scope, standardize, leave things out

●  Think “shift left”

●  Don’t overbuild!

Must See Presentations

●  Charlie Sharpsteen on performance tuning, 1:30PM Terrace Salon

●  Martin Jackson on collaboration culture, 11:15AM Friday, Grand Hall

●  Russ Mull and Zack Smith on Puppet HA, 2:30PM Friday, California room

●  Multi-tenant station in the Exhibit Hall