SNEAK PEEK The Puppet Environment at Fairfax Media / Domain Group

Dejan Golja // Gonzalo Servat

Introduction

Dejan GoljaSenior Systems Engineer // Fairfax Media

Twitter @dejangolja

Introduction

Gonzalo ServatSenior DevOps Engineer // Domain Group

Twitter @gservat

Who are we?

Our Puppet History

☁ Early Puppet adopters … since version 0.25.X

☁ Large scale environment with distributed team

☁ We made every possible mistake

We’re on our 3rd major Puppet implementation!

Lessons Learned

☁ Do NOT make changes directly on the Puppet Master

☁ DO your testing - lint, code validation, etc.

☁ Do NOT pollute your downloaded official modules

☁ DO define and document the standard workflow

☁ Do NOT store sensitive data into modules

Example of ugliness (old site.pp)

Standardised process for deploying changes

Easily repeatable for multiple

environments

Puppet “V3” Requirements

High Availability

Bring order to our modules

Ingredients

☁ Packer

☁ Puppetlabs R10k

☁ Custom Ruby Service - “Captain Hook”

☁ Notification Systems - Slack/Flowdock

EC2 ASG ELB

SNS SQS RDS

CloudFormation

☁ Module preferences☁ Puppetlabs supported☁ Vendor supported☁ Puppet approved

☁ Create module guidelines else it gets very messy!☁ Each module lives in a separate Git repository

Puppet Modules

Git Flow

Feature Branch

Workflow

Architecture

Puppet Client

Puppet CA

Puppet Masters

Puppet DB

Architecture

$ aws cloudformation create-stack \

--stack-name puppetmaster-staging \

--templatebody file://infrastructure-puppetmaster.template \

--parameters \

ParameterKey=Environment,ParameterValue=staging \

ParameterKey=InstanceType,ParameterValue=m3.large \

ParameterKey=MinScalingGroupSize,ParameterValue=2 \

ParameterKey=MaxScalingGroupSize,ParameterValue=4

CloudFormation Example

Awesome, so we have a neat-looking HA/FT set-up…

…but how do we manage pushing changes to Puppet Masters?!

☁ Puppet masters can “come and go” randomly

☁ Keep the modules up to date per environment

☁ K.I.S.S. - Tame the learning curve for the team

☁ Avoid reinventing the wheel

Challenges

OUR SOLUTION

P3 Workflow

1. Make changes and commit/push to git server2. Git server triggers post-commit hook (POST) to Captain Hook server3. Captain Hook server validates the payload & creates new message:

☁ Full refresh ➩ r10k deploy environment -p☁ Light refresh ➩ r10k deploy environment☁ Module refresh ➩ r10k deploy module <name>

4. Captain hook server pushes new message to SNS5. Captain hook client polls & reads messages ( ➩ r10k)6. … and we get notified in slack/flowdock/dashboard

Typical Puppet Workflow

DEMO

☁ Fairfax is a large scale complex environment

☁ Many systems engineers are constantly pushing changes

☁ Puppet architecture designed for HA and fault tolerance

☁ Puppet workflow helps us deploy changes to multiple

masters safely and easily

Summary

Our Blogshttp://engineering.fairfaxmedia.com.au/

http://tech.domain.com.au/

Our Codehttp://github.com/fairfaxmedia

Your Feedback or QuestionsYou can contact us on twitter: @dejangolja @gservat

Before you go ...