2 0 0 , 0 0 0 L I N E S L AT E RO U R J O U R N E Y T O M A N A G E A B L E P U P P E T C O D E

D AV I D D A N Z I L I O C O N S TA N T C O N TA C T

W H O A M I ?

• Puppet Evangelist at Constant Contact

• Joined Constant Contact in 2014

• SysAdmin background but more of a developer these days

• Second time speaking at PuppetConf

W H AT I S C O N S TA N T C O N TA C T ?BACKGROUND

• Founded in 1995

• Headquarters in Waltham, MA

• Offices in San Francisco, NYC, London, among others

• ~ 1,500 Employees, ~ 350 developers, ~ 70 operations

• 160+ apps (SOA - microservices for hipsters)

• We’re hiring! (See me after for details)

W H AT ’ S T H I S TA L K A B O U T ?

• An overview of our environment and our challenges

• Our successes, failures, and lessons from trying to transform Puppet in a large enterprise

• Plans for achieving the Puppet singularity

• My quest to understand and affect change in a massively complex organization

R U N N I N G P U P P E T AT S C A L E I S H A R D

TL ;DR

A L O N G T I M E A G O I N A G A L A X Y FA R FA R A W AY

E A R LY A D O P T E R S Y N D R O M E

• Started using Puppet in 2009 with version 0.24.8

• You name the bug, we’ve probably seen it

• Forge didn’t exist so we had to write everything

• Puppet talent was (still is) hard to come by

• Writing Puppet was a little harder back then…

Feature 0.23.x 0.24.x 0.25.x 2.6.x 2.7.x 3.x 3.2.x 3.4.xDynamic Scope X X X X X      Appending to attributes in class inheritance (+>) X X X X X X X XMulti-line C-style comments   X X X X X X XArrays of resource references allowed in relationships   X X X X X X XOverrides in class inheritance   X X X X X X XAppending to variables in child scopes (+=)   X X X X X X XClass names starting with 0-9   X X X        Regular expressions in node definitions     X X X X X XAssigning expressions to variables     X X X X X XRegular expressions in conditionals/expresions     X X X X X Xelsif in if statements       X X X X XChaining Resources       X X X X XHashes       X X X X XClass Parameters       X X X X XRun Stages       X X X X XThe “in” operator       X X X X X$title, $name, and $module_name available in parameter lists       X X X X XOptional trailing comma in parameter lists         X X X XHyphens/dashes allowed in variable names *         X      Automatic class parameter lookup via data bindings           X X X“Unless” conditionals           X X XIteration over arrays and hashes             X XThe modulo (%) operator             X X$trusted hash               X

• Lots of the code we wrote back then is still around…

• The company was growing so fast, nobody had time to focus on Puppet

• Ops mandated that all configuration and app deployment had to be done with Puppet

• Lots of push back

• Puppet became a dirty word

T H E G O O D

• Everything is in Puppet

• Everything is in Git

• Our infrastructure is built to be ephemeral

• We can provision physical nodes about as fast as we can spin up a virtual machine

• Moving an app to OpenStack is just a matter of running Puppet

T H E B A D

• Stuck on Puppet 2.7.3; lots of the code was written before parameterized classes

• Most of our code lives in a single Git repo

• Testing is up to the developer

• No design beyond the initial implementation; most of the code just evolved organically

• Propensity for home-grown solutions, and a habit of thinking only in terms of the current use case

T H E U G LY

• Massive codebase (more on this in a second)

• Frightening complexity

• Tightly coupled codebase (massive understatement)

• Very few tests (effectively none)

• Very little documentation

• Too easy to get into production

• No dependency management

F I L E S L A N G U A G E B L A N K C O M M E N T C O D E8 , 2 6 7 P U P P E T 4 3 , 0 5 9 2 1 , 9 3 3 1 9 8 , 9 0 11 , 0 7 6 E R B 2 5 , 1 9 8 6 6 1 4 3 , 7 2 4

3 9 1 B O U R N E S H E L L 6 , 7 6 1 6 , 5 3 1 5 5 , 1 6 21 9 3 X M L 2 , 7 1 4 5 , 1 1 1 3 7 , 3 6 05 8 P E R L 4 , 1 9 9 6 , 5 2 0 2 3 , 8 6 6

5 4 2 R U B Y 4 , 9 2 2 1 , 7 2 0 2 2 , 6 6 85 1 P Y T H O N 2 , 9 5 4 1 , 6 2 2 1 1 , 2 6 86 7 S Q L 4 5 2 1 9 7 9 , 9 9 9

4 6 6 YA M L 6 2 3 8 4 2 8 , 7 9 91 0 0 O T H E R 2 , 5 3 9 6 , 0 3 4 8 , 3 2 8

1 1 , 2 1 1 T O TA L 9 3 , 4 2 1 5 0 , 5 7 6 5 2 0 , 0 7 5

F I L E S L A N G U A G E B L A N K C O M M E N T C O D E2 , 3 7 6 R U B Y 5 2 , 3 5 2 3 0 , 4 6 3 2 2 9 , 6 5 4

4 0 E R B 6 1 7 1 1 2 , 2 7 17 X M L 5 0 2 1 , 5 5 4

5 6 YA M L 1 6 4 1 , 5 1 13 2 B O U R N E S H E L L 4 0 2 4 7 9 1 , 6 3 81 6 J S O N 7 5 0 1 , 1 7 58 2 P U P P E T 1 5 9 4 6 6 9 42 3 S M A R T Y 3 8 0 3 6 92 L I S P 4 4 9 6 3 0 0

1 4 O T H E R 1 3 3 1 1 3 6 2 92 , 6 4 8 T O TA L 5 3 , 8 8 6 3 1 , 2 1 4 2 3 9 , 7 9 5

T H E J O U R N E Y B E G I N S

U N D E R S TA N D

I D E N T I F Y O U R U S E R S

• Consumers use Puppet to deploy their apps. They just want Puppet to work.

• Developers may write Puppet modules or profiles. They may consume Forge modules.

• Architects help shape the strategic direction of Puppet in the organization.

• Consumers have a very different use case than Architects or Developers.

• Most of our users are Consumers, so we should focus on them.

• Consumers aren’t necessarily interested in learning Puppet, but they need to use it to deploy their apps.

• Interfaces are very important to these people.

T E A C H

T E A C H I N G P U P P E T

• Most people working with Puppet didn’t really know the language well

• But they had to get shit done

• Rampant propagation of bad patterns

• Wrote a 4-hour intro class and started teaching

• Get everybody on a common baseline, a good foundation to build good habits

• Offer more advanced training to people who want it

U P G R A D E

A M O D E R N V E R S I O N O F P U P P E T

• Puppet 2.7.3 is buggy and slow

• Couldn’t use any modules from the Forge out of the box because of trailing commas

• Puppet 2.7 was holding us back

• Offload file serving to Puppet 3 masters, cheap performance win: bit.ly/1VfSwYM

• Problem: code isn’t compatible with Puppet 3

classmysql_site{

#Setsomedefaults$mysql_log_rotate_enable=0$mysql_log_rotation='daily'$mysql_log_retention='14'$mysql_slowlog_rotate_enable=0$mysql_slowlog_rotation='weekly'$mysql_slowlog_retention='4'$mysql_errlog_rotate_enable=0$mysql_errlog_rotation='monthly'$mysql_errlog_retention='6'$mysql_genlog_rotate_enable=0$mysql_genlog_rotation='daily'$mysql_genlog_retention='14'

...ANDSOON…

}

classmysql_site::devinheritsmysql_site{

#OverridesforDevenvironment$mysql_log_rotate_enable=1$mysql_slowlog_rotate_enable=1$mysql_errlog_rotate_enable=1$mysql_genlog_rotate_enable=1

}

classmysql_site::dev::appinheritsmysql_site::dev{

#Overridesforappindevenvironment$mysql_slowlog_rotation='monthly'$mysql_errlog_rotation='daily'

includemysql

}

M I N I M U M V I A B L E R E FA C T O R

• We knew the code was bad, but we didn’t have the resources to focus on a redesign

• Refactor just enough to get to Puppet 3

• Fix templates

• Eliminate dynamic scope

• Hiera played a big role in this

• Move hosts to Puppet 3 as code becomes compatible

TA K I N G T H E P L U N G E

• We built out a Puppet 3 infrastructure alongside the legacy infrastructure

• Shared CA, so the clients could talk to both sets of masters

• Move hosts over as the code becomes compatible

• Turns out this is really hard when there’s no standard way to classify hosts

• We started with the easiest stuff: our own hosts

• Moved on to our CD apps

• In process of moving legacy apps now

M E A N W H I L E …

• We couldn’t stop time and do all this work in a vacuum, life was still happening around us

• New modules are put into their own repos and deployed with r10k

• We’ve kept Puppet 3 environment up to date. Started on 3.4.3, now on 3.8.3

• New modules must have tests and documentation

• We’re using roles and profiles for new things

• CI workflow and infrastructure

• Testing environments, harnesses, and helpers

• Deployment tooling

• Knowledge of testing frameworks

• No official images

N E X T S T E P S

T H I N G S W E ’ R E TA C K L I N G N E X T

• Node classification

• Roles and Profiles

• Decoupling our code

• Desire for bespoke everything

• Writing modules with a single entry point

• Dependency management

• The Forge test

T H E F O R G E T E S T / P U P P E T A P P R O V E D T E S T / D E F I N I T I O N O F D O N E

• Can this module be deployed to the Forge?

• Is there any proprietary data embedded?

• Is it modular enough to meet unforeseen use cases?

• Does it have a robust suite of tests?

• Does it conform to community style expectations?

• Is it fully documented?

I T ’ S A L L A B O U T T H AT I N T E R FA C E

• Make it easier to use our modules

• Our consumers care about the interface, not the implementation

• Design better and fewer interfaces

• Modules with well defined interfaces for composability and better dependency management

• Documentation and tests as part of the definition of done

T E S T I N G

• Make testing easier

• Make testing automatic

• Make testing a part of the culture

• Teach people about BDD

• Show them how much easier and more reliable an automated test can be than a manual one

• Eventually, make testing required

L E S S O N S L E A R N E D

• Know your audience

• Don’t try and please everybody

• It’s always going to take longer than you think

• Don’t forget about the supporting infrastructure

• People, process, and technology. In that order

• It’s never going to move as fast as you want it to. Walk, don’t run

• Don’t let yourself become paralyzed by complexity

• Take notes, write down things you need to work on, you’ll get to them eventually

• Always take the easy wins

• Focus on high impact improvements, help as many people as you can

• Small improvements add up

There’s nothing more successful than success

L E S S O N S T O B E L E A R N E D

• How do we get people to feel invested in the design?

• How can we reenforce good behaviors?

• How can we make sure our massive codebase is being regularly evaluated?

• How can (should?) we hold people accountable for their Puppet code?

• How can we get operations folks to realize they’re software developers?

T H E E N D

H O W T O F I N D M E

• @djdanzilio on Twitter

• danzilio on Freenode (you can usually find me in #puppet-community, #puppet-dev, and #puppet)

• danzilio on GitHub and The Forge

• david (at) danzil (dot) io

• blog.danzilio.net

I M A G E C R E D I T S

• The space photos are from the NASA Image Galleries and are either in the public domain or licensed under CC-BY 2.0

• All other photos are mine or my employer’s