Killer R10K Workflow

Automating the KillerRobots, all 10K of Them

We all love to automate things. The very fact that you are here at PuppetConf and attending this talk leads me to believe you like to automate things. !When it comes to puppet adoption, I think everyone goes through various phases of automation and maturity. First you learn how to write puppet modules, and how to incorporate puppet forge modules. !You might be using roles and profiles to help structure and organize your puppet code. In addition, you may also be using hiera to help externalize and organize your configuration data. What you’re focusing on, and rightly so, is getting all the things in place so that puppet can help manage your infrastructure

and kill all the snowflakes. !At some point it hits you - how am I going to get all my puppet code deployed in an automated, repeatable way? After all, this is no time to start using manual methods in your process. !At its essence, r10k helps address this very question. How many of you are currently using r10k? !R10K is a key component of our puppet deployment automation at Time Warner Cable. In addition to r10k, we utilize other tools such as Jenkins, Capistrano and Sinatra in our automated workflow. This workflow supports iterative development as well as production deployments for our puppet code. As you

will see, we had to learn how to crawl a bit before we starting running. !An important benefit of our workflow is that it allows people to focus on writing good module code and not worry how their changes will get deployed. It just works. This is no different than an application developer who simply wants to write good code and not worry how the war will get deployed. !I just learned last night that the Jenkins project is using a version of the first post-receive hook I published!! Very Cool !It’s the end of the day and I know everyone’s brain is full. Hang in there - we can do this! !!

Who Am I@phil_zimmerman

- Senior Software Engineer at Time Warner Cable - Part of a team of full-stack engineers doing devops-y things - Get to work with some cool tools like

- Puppet, Jenkins, Artifactory, Docker, Packer, OpenStack, Graphite, Sensu, etc

Early Days

R10K Workflow Awesomeness

Demo Time

- Early Days - First Processes - Learning what worked/what didn’t - Led to current processes and tools !

- R10K Workflow Awesomeness

- Dynamic branch creation

- Dynamic Puppetfile manipulation

- R10K and release creation

- Production deployment !- Demos

- Workflow - Understand your job - Understand your tools - Don’t think about it anymore - What’s not in this image

- automate - monitor - improve

The Journey of A Thousand Miles Begins

With A Single, Monolithic Repo

- All internally-built puppet modules and hieradata in one repo

A Single Repo? What the F*@K?!!

- The horror - Yes, Gary Larizza is yelling - Who would do such a thing??

Why Put Everything in a Single Repo?

Simplify Development

Easy Jenkins Flow

Puppet Code and Hiera Data Together

- (click animation) SIMPLIFY DEVELOPMENT - One repo is all you need to make changes - All Puppet code in one place !

- (click animation) EASY JENKINS FLOW - Just a few jobs needed for CI, tests and deployment !

- (click animation) PUPPET CODE AND HIERA DATA TOGETHER - puppet code and hiera data tagged/deployed together - easy to test and deploy changes (config data in sync with code)

Just Starting

Out

- New things to grok - Puppet Enterprise - New module development - Integrate Hiera for Data Separation !

- Make it easy for people to start using puppet - Had to get this in place quickly

The toolset:

- Jenkins

- Github

- Capistrano: to get puppet code on the masters. Chose it for its ability to

run commands on multiple remote servers at once.

- didn’t want to force mcollective

- Hipchat: IM notifications

Single CI Job • rspec-puppet • syntax check • lint

Single Release Job • create/push tag

Single Deploy Job • Capistrano tasks

• poor man’s dynamic environments • kludgy git logic w/conditionals in Capfile

For All Modules

- Single CI Job - tests, syntax check and linking for every module in repo !

- Single Release Job - simple to tag single repo and push to Github !

- Single Deploy Job - select tag to deploy and pass it as arg to Capistrano task

Forge Modules

Capistrano -> Puppet Module Tool

Worked Well… Until It Didn’t

- Job to manage forge modules - parameterized: forge module name - Capistrano wrapping ‘puppet module tool’ calls - worked well for a while

Did I mention the

Capistrano tasks?

KLUDGY

- Yes, yes I did. Let’s take a look at an example and hope your eyes don’t burn too bad

task "create_puppet_env", :roles => :puppet_master do if exists?(:branchname) run "if [ -d #{environment_basedir}/#{branchname} ]; then cd #{environment_basedir}/#{branchname} && git pull origin #{branchname} ; else \ git clone #{module_repository} #{environment_basedir}/#{branchname} --branch #{branchname} ; fi" else puts "Please provide a valid git branch name as an argument" end end

- Task to create a “dynamic environment” - What, you can’t read that? Here (click)

task "create_puppet_env", :roles => :puppet_master do if exists?(:branchname) run "if [ -d #{environment_basedir}/#{branchname} ]; then cd #{environment_basedir}/#{branchname} && git pull origin #{branchname} ; else \ git clone #{module_repository} #{environment_basedir}/#{branchname} --branch #{branchname} ; fi" else puts "Please provide a valid git branch name as an argument" end end

- (Sarcastic) Oh yeah - that’s SO much better ;-) - I can’t believe I ever thought this was a good idea :-)

- Works, but it’s ugly - way too much logic - embedding git commands - Kind of makes your eyes burn doesn’t it?

Upgrade Forge Modules

Version Management == Face Palm

Upgrading Affects all Environments!

- No tie between version of internal modules and forge modules - Release management became really hard

- Upgrading a forge module affects all environments - forge modules in own module path on masters - no concept of “environments”

- Couldn’t test without impacting other envs, especially production

Let Me Count the Other Ways…

This Started to Fail

Simple Comes at a Price

You mean I have to wait for tests

to run for ALL modules before I know if my changes are good?”

“I only changed one module….

- Runs ALL tests in ALL modules - fixtures and test setup - execution time

- More modules == more tests == longer wait times

I have to deploy

everything in order to get my changes on the masters? Sigh”

“I only changed one module….

- Impossible to simply deploy a single module - all other modules and hiera come along for the ride

everything just to get my hiera data on the masters?”

“I need to change some hiera data…. I have to deploy

- just want to deploy a small hiera config change - I have to deploy the entire repo to get my small change onto the masters

I’m Losing

My Patience!

- This gets old pretty quickly - all the waiting - having to deploy everything, even all the things I did not change - the monolith is starting to feel heavy and burdensome

Oh - and that little problem where

upgrading forge modules can break production…

Stop the Madness !!

- Given all the shortcomings, this setup is no longer meeting our needs - We need to make some adjustments and work smarter - (click animation)

- grab a drink of water and catch breath

Recap of Early Days:Monolithic RepoLong CI CyclesAll-Or-Nothing DeploysUpgrading Forge Modules

- Single repo for all modules and hieradata - Runs all tests for all modules in repo - I just want to deploy my tomcat change (nope) - Upgrading forge modules was hard

- no tie between forge module versions and internal module versions

- We came up with a better way of doing things. - kill the monolithic repo - added a couple tools - improve the process

Tools Can Be

Awesome

(we just needed a couple more)

Make Them Work For You

-

R10K

R10K

- Same tools as before with some additions: - ruby - sinatra - post-receive hook - and of course r10k

R10K!

https://github.com/adrienthebo/r10k

- Adrien Thebo - “finch” - help automate deployment of puppet modules and environments - Puppetfile format and native implementation of dynamic environments - NOTE: this is a server-side tool that runs on all puppet masters

Deploys Puppet Code

Handles Git/Svn Fu

Is Awesome

- R10K deploys ALL puppet modules - internal AND forge

- Handles all internal git/svn management - caching - removes stale modules

- And yes - it’s awesome !

R10K and Puppetfile

Match Made in Heaven (or Portland)

Manage Module Versions

Inventory of Puppet Environment

Puppetfile

Format

- You are probably already familiar with the Puppetfile format, but it’s worth a quick review

Inventory of Modules and Their Versions

Puppetfile lives in its

own repository

Where we define all of our internal and forge modules

- module name

- where to get it

- version

Meta inventory of our puppet setup

Puppetfile lives in its own

Repository

- This is significant to note - puppet environments represent branches in the Puppetfile repo - I’ll talk about how Puppetfile branches get created soon

Inventory of Modules and Their Versions

- Provides the tie of all module versions for a given puppet environment - The monolith is dead and gone!

Each module in its own

repository AND r10k

R10KDeploy

- Many options to deploy modules and environments - deploy the world (every module in every environment)

- use sparingly - very time-consuming - deploy a single environment (every module in a puppetfile for a given branch)

- common use-case - we use all the time - deploy a single module (tomcat or nginx)

- common use-case - we use quite often

r10k deploy environment test -p

deploys all modules in Puppetfile for the test branch

r10k deploy module tomcatdeploys a single module!

- A couple examples (click animation) !- deploy a whole environment (all modules in the Puppetfile repo’s test branch) !- deploy a single module (one module in a pre-existing environment)

CI Job Per Module • rspec-puppet • syntax check • lint

Release Job Per Module • create/push tag • select module from dropdown list

Deploy Job For Each Module And Hiera • simpler Capistrano tasks • wrap r10k calls to each master/node

- Jenkins revisited now that we have each module in its own repository - !- CI Job per module

- tests, syntax check and linting !- And, as you might expect, there is a Release Job per module

- simple to tag a module repo and push to Github !- Deploy Job for each module and hiera, which is in its own repo

- we now have independently releasable modules - simple Capistrano tasks

desc "for specified branch in puppetfile repo, use r10k to deploy all modules for the specified environment." task "update_environment", :roles => :puppet_master do if exists?(:branchname) run "r10k -v debug deploy environment #{branchname} -p" else puts "Please provide a valid git branch name as an argument" end end

- Capistrano tasks now simply run r10k commands - no more server-side logic or conditionals - here is an example of a Capistrano task that uses r10k to deploy a given environment

- much cleaner and easier to follow what the task is doing

Puppetfile Manipulation and Branch Creation

Before r10k can deploy an environment:

- the puppetfile repo branch has to be created and pushed

- since r10k runs on each master, it will sync and reference the puppetfile repo on each master as well

- how does the puppetfile branch get created and pushed to the remote?

Now consider iterative development

- how do my changes get deployed on subsequent pushes to the same branch? !Finally, how do I know when r10k is finished deploying my changes? Let’s look at an example!

mod 'tomcat', :git => 'git@github.webapps.rr.com:fylgia/tomcat.git', :ref => 'RELEASE_1.0.13'

Tomcat Example

tomcat entry

- production Puppetfile branch

- current version of tomcat in production is RELEASE_1.0.13 !I need to add some functionality to the tomcat module - here’s the workflow

go to tomcat dir (production branch) !

‘git checkout -b dev_change_foo’ ‘git push origin dev_change_foo’

On your local workspace:

- go to tomcat module repository (or clone it if need be)

- create a branch called dev_change_foo and push it to the remote

Create the ‘dev_change_foo’ branch

in Puppetfile repo

When I push the ‘dev_change_foo’ branch for the tomcat repository:

- A branch of the same name gets created in the puppetfile repository

- Dynamic Puppetfile branch creation!

Change the :ref for tomcat to

‘dev_change_foo’

In the ‘dev_change_foo’ branch of the Puppetfile repo:

- update the tomcat :ref to ‘dev_change_foo’

- push the ‘dev_change_foo’ branch to the remote !Dynamic Puppetfile Manipulation!

- auto update module ref to point to the same branch name

- this effectively associates my module’s ‘dev_change_foo’ branch to the puppet env of that name

Call r10k to deploy ‘dev_change_foo’

Notify Me When It’s Finished

- the ‘dev_change_foo’ environment is now on all my masters, with my latest tomcat changes - I’d like to get a notification when r10k is finished deploying my changes

Yes - the Jackie Chan face - I know some of you might have this feeling :-) !How did all of this happen?

Explain the Magics

Hint:

All of this automation is handled by a sinatra post-receive git hook. This hook is a web app that listens for post events to a defined endpoint. !Github web service hooks post a JSON payload that provide information regarding the git push that triggered it. Inspecting the payload shows what kind of push triggered it:

- create branch

- delete branch

- etc !Each puppet module needs to have the hook url configured in its service hook settings so that it gets called on every git push. The hook recognizes 3 git push types and, as you’ll see, does a lot of work.

Create Branchr10k deploy environment

If a new module branch is pushed, the hook will:

- create new puppetfile branch of the same name

- for the module that was pushed, set the :ref to be the name of the branch

- commit and push the new puppetfile branch

- call ‘r10k deploy environment branch_name -p’

- notify a configurable endpoint (like hipchat) with r10k output

Modify Branchr10k deploy module

If a change is pushed to an existing module branch, the hook will:

- checkout the puppetfile branch of the same name

- ensure the :ref of the module is still the name of the branch

- call ‘r10k deploy module tomcat -p’

- notify a configurable endpoint (hipchat) with r10k output

Delete Branchauto-delete Puppetfile branch

If a module branch is deleted from the remote, the hook will:

- delete the puppetfile branch of the same name from the remote (auto cleanup)

- notify a configurable endpoint (hipchat) noting the puppetfile branch was deleted

- the next time r10k runs, the deleted puppetfile branch will be removed from the masters

- r10k’s built-in ability to purge environments

Testing Multiple Modules in the

Same Environment

For example:

- tomcat module (add a new web context, for example)

- nginx profile module (maybe I need to add a new resource location)

So, I need to make changes to the tomcat and profile modules and test them in the same environment

go to profile dir (production branch) !

‘git checkout -b dev_change_foo’ ‘git push origin dev_change_foo’

From our original tomcat example, we already have a branch called ‘dev_change_foo’

- now just create the same branch in the profile module and push it

- the post-receive hook will checkout the puppetfile ‘dev_change_foo’ branch

- then the :ref for the profile module will be ‘dev_change_foo’, the branch will be pushed, then r10k deploy will take care of the rest.

- here’s what the resulting Puppetfile entries will look like:

mod 'profile', :git => 'git@github.webapps.rr.com:fylgia/profile.git', :ref => 'RELEASE_0.1.124' mod 'tomcat', :git => 'git@github.webapps.rr.com:fylgia/tomcat.git', :ref => 'RELEASE_1.0.13'

production

mod 'profile', :git => 'git@github.webapps.rr.com:fylgia/profile.git', :ref => ‘dev_change_foo’ mod 'tomcat', :git => 'git@github.webapps.rr.com:fylgia/tomcat.git', :ref => 'dev_change_foo'

dev_change_foo

When r10k runs, the ‘dev_change_foo’ environment will have all modules deployed at their production versions, except for tomcat and profile. These modules will be at the tip of their ‘dev_change_foo’ branches. !Each subsequent push will force r10k to run again, thus updating the module to be at the tip of the ‘dev_change_foo’ branch.

Truly Dynamic Environments!

The beauty of this is I know that each time I push changes, they will be automatically deployed/updated on the puppet masters !This is perfect for iterative development and testing.

The Post-Receive Hook

reaktor

The middleware at the center of all this is reaktor. It is a modular rack app that I recently open-sourced. Utilizes queueing to handle the incoming create/modify/delete requests. !Reaktor

- dynamic Puppetfile branching

- drives r10k

- provides notifications !

Default SetupGitHub or GitHub Enterprise

Hipchat

Just provide some config!

Default Implementation

- github and github enterprise

- hipchat

- you supply the config

REAKTOR_PUPPET_MASTERS_FILE

REAKTOR_HIPCHAT_TOKENREAKTOR_HIPCHAT_ROOMREAKTOR_HIPCHAT_FROM

PUPPETFILE_GIT_URL

Environment variables

REAKTOR_PUPPET_MASTERS_FILE - Tells Capistrano where to call r10k !PUPPETFILE_GIT_URL - needed for dynamic puppetfile branching !Various hipchat variables needed for hipchat api !

Other Chat ProvidersCampfire, Slack, etc

If it has an API, it’s pluggable!

Easy to plug in a different chat option

- slack

- campfire !- Implement a well-known interface

- Contribute back so others can use it!

Other Git ProvidersGitlab, Bitbucket, etc

Need to Determine Best Approach

Can work with different git providers with some effort

- support custom web hooks

- payload needs enough meta data to determine what action is being taken ! - gitlab and bitbucket don’t provide meta-data needed

- approach needs to be thought out !Shameless plug to contribute and help make it more awesome !Now let’s take a look at our r10k workflow and how it’s used in production deployments (or deployment to any static environment).

Create Release

Deploy - Ship It!

- As you’ve seen, we utilize Jenkins pretty heavily

- Create Release and Deploy are the 2 jobs used in the release/deploy workflow

Create ReleaseModulefile

RELEASE_1.0.13 -> RELEASE_1.0.14

versionfile (hieradata)

Parameterized job

- drop down list of modules and hieradata !Calls ruby script

- increment version in the Modulefile or versionfile

- create tag

- commit and push tag/updated file

DeployParameterized Job

environment

version to deploy

2 Parameters

- static env to deploy to (test/production/whatever)

- version to deploy (auto-generated list of tags)

Each module has a deploy job

- use a Jenkins “seed job” template to create all the deploy jobs

Puppetfile Manipulation (again)

change :ref to selected version

r10k deploy selected environment

commit/push updated Puppetfile

Calls ruby script

- modify Puppetfile

- sets module :ref to the new version

- commit/push Puppetfile

Calls Capistrano task

- call r10k deploy environment (test or production)

One-Click Production

Deploy

Production Deploys made easy!

Production Deploys made easy!

Workflow Recap:R10K and PuppetfileEach Module in Own RepoPost-Receive Hook GoodnessProduction Deployments

- r10k and Puppetfile (match made in Heaven) - Independently Releasable Modules and Hieradata - Post-receive hook:

- dynamic Puppetfile manipulation and branch creation - notifies when r10k is finished deploying changes

- r10k used in production deployments

I’m going to show some r10k workflow with dynamic environments and I’ll exercise all 3 event types that the post-receive hook implements (create branch, modify branch, delete branch). !First let’s start up the post-receive app by running ‘rake start’. Now I’ll go to my local workspace in the tomcat module and create a new branch called ‘dev_r10k_demo’. I’ll add a notify block just saying to let us know it’s running against the dev_r10k_deme branch. Once I push the branch to the remote, the

hook starts working. You’ll see the stdout, as well as hipchat notifications with the relevant output.

While that’s running let’s go to the Puppetfile repo and do a ‘git pull’. You will see that we now have a dev_r10k_demo branch. As we open up the Puppetfile in this branch, notice that the tomcat git ref is now pointing to the dev_r10k_demo branch as well. !Going back to the hipchat room, it shows that r10k has finished deploying the dev_demo environment on the masters. Back over to hipchat, we can see that r10k has finished running, and we have a shiny new dev_r10k_demo environment. !Now I’ll go to a vm to test against my new environment. I’ll perform an actual puppet agent run against the new environment in noop mode. It’s thinking. Still thinking….. Still THINKING…. !Ok - and now we can see our notify string letting us know we are running against the dev_r10k_demo branch.

Back over to my local tomcat workspace, I’m finished with the branch so I’ll delete it. Again, you can see the hook firing as shown by the standard out in the log. Over in hipchat, we can see that the branch was deleted from the puppetfile repo. We can check this locally as well, by running ‘git fetch —prune’

and seeing that the dev_r10k_demo branch was deleted.

Starting out in the puppetfile repo, I have the test branch checked out. Let’s run a git pull to ensure I have the latest, then open up the Puppetfile. You’ll notice that the tomcat module is at version RELEASE_1.0.19. !Now we will switch over to the tomcat module repo. We will do a pull and then cat the Modulefile. Notice that the version here is also at RELEASE_1.0.19. !Now we will switch screens and take a look at the Jenkins Create Release job. Clicking on “build with parameters” it brings up the drop-down list of modules. We will choose tomcat and click “Build”. This process is very quick. Going up the console, you’ll see the output showing that the original version is

RELEASE_1.0.19, and where it was updated to RELEASE_1.0.20. Back at the bottom you’ll notice where it pushed a new git tag called RELEASE_1.0.20. Over to hipchat, we can see that the changes indeed are reflected in git. Back to my local workspace in the tomcat module repo, we can pull and see that

the Modulefile is now at version RELEASE_1.0.20. !Now we’ll switch over to the tomcat deploy job. Clicking on “Build with Parameters” shows the ENV we can choose to deploy to, along with the version of the tomcat module. The first version that shows up is the one we just created. Clicking on the version shows all the available tags we can choose from to

deploy. We will deploy the 1.0.20 version. Clicking on the ENV dropdown shows the 2 static environments we have, test and production. We will deploy to test. We click on the Build button and then bring up the console output so we can see what’s going on. The deploy jobs are running ‘r10k deploy

environment test -p’, which deploys the test environment to multiple puppet masters. You can see all the modules being deployed in the test environment. Jumping back to hipchat, again git is showing that the puppetfile was updated in the test branch. And we have success. !Let’s jump back over to my local puppetfile checkout and do a git pull. Now we open up the Puppetfile again and notice that the tomcat git ref is now at version RELEASE_1.0.20. !That ends the create release and deploy demonstration.

Conclusion:Early DaysR10K Workflow AwesomenessDemos

- I’ve shown some of the early processes and how those worked for a while, but started to fall short. - We saw how some of the learnings drove improvements in our processes - R10K makes up a central part of our puppet deployment process - I’ve shown how we’ve adopted r10k and added some additional automation to achieve dynamic environment and production deployment flow - Demonstrations helped bring all the info together and provided examples of our workflow in action. - r10k is continuing to improve and adapt. Some of what the current post-receive hook may be replaced by native capabilities. - However, the need to drive r10k and receive notifications are core aspects of the hook that will still be valuable moving forward. !- if not using r10k or not using repo per module, now a great time to start - Thank Adrian and Gary

Helpful Links and References

https://github.com/adrienthebo/r10k

http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/

http://garylarizza.com/blog/2014/03/07/puppet-workflow-part-3b/

http://philzim.com/2014/05/02/an-automated-r10k-workflow-that-works/

https://github.com/pzim/reaktor