puppet conf 2013
TRANSCRIPT
CONTINUOUSLY INTEGRATED PUPPET IN A DYNAMIC ENVIRONMENTSAM BASHTON, BASHTON LTD
CONTINUOUSLY INTEGRATED PUPPET IN A DYNAMIC ENVIRONMENTMASTERLESS PUPPET: WHY AND HOWSAM BASHTON, BASHTON LTD
ABOUT MELinux guy since Slackware, floppy disks and root + bootUsing Puppet since 2007Run a company in Manchester, North West EnglandWe provide outsourced ops for other companies
OUR FULLY MANAGED ENVIRONMENTSPrimarily transactional websites (ecommerce)Majority (70%+) on Amazon Web Services (AWS)Majority using CentOS
HOW WE WORKSimple is better than complexComplexity is worth adding only if it provides obvious functional benefits
ReusabilityResilience
WHY DID WE PICK AWS?
Featureset and toolset massively in advance of any other cloud provider, public or private#1 customer reason for switching to AWS? The ability to scale on demand
TOOLS WE USE FOR BUILDING AND MANAGINGDo one thing and do it well
CloudFormation Amazon tool to manage infrastructurePuppet Manage system configurationPulp centralised repository, manages package revisionsJenkins
HOW WE USE PUPPETNo PuppetmasterPuppet manifests, hieradata and modules distributed to all machines via RPMAll machines boot with a common, blank image and get configured at first boot
WHAT'S WRONG WITH MASTER BASED PUPPET?Pets vs Cattle
Puppet designed for a world of servers as petsWe do not live in that world
PUPPET DESIGNED FOR PETSMany assumptions in Puppet presume that your servers are petsSome of these work against us when managing a herd
MANUAL CERTIFICATE SIGNINGClearly unsuitable when machines are automatically provisioned
POTENTIAL WORKAROUNDS:
AutosignUse/write another automated certificate generation mechanism
Possibly tied in with autoscaling
NO MECHANISM FOR CLEANING OLD HOSTSLikely to have hostnames reused, causing machines to fail to configurePuppetmaster will fill with certificates for machines that ran for a few hours and went away again
POTENTIAL WORKAROUNDS:
Use UUID certificatesAgree not to look in the certificate directoryWrite mechanism for cleaning up old certificates
HOSTS CONFIGURED BASED ON HOSTNAMEOur machines have names like ip172265123How does Puppet know what type of machine this is?
POTENTIAL WORKAROUNDS
Use an external node classifierUse some mechanism for giving a better hostname, eg web172265123 and use regex for nodenames
PUPPETMASTER IS A SINGLE POINT OF FAILUREIf the Puppetmaster fails, we can no longer autoscale upIn particular, this could be a problem if there is availability zone failure
POTENTIAL WORKAROUNDS
Clustered Puppetmasters
WORKAROUND RECAPUse/write alternative certificate management softwareWrite an external node classifier / mechanism for setting hostname appropriatelyCluster multiple Puppetmasters
WHAT WE DID INSTEADDecided using a Puppetmaster was trying to fit a square peg into a round holeInstead, decided to run Puppet without a master
APPLYING LOCAL PUPPET MANIFESTSpuppet apply --modulepath=/etc/puppet/modules example.pp
DISTRIBUTING MANIFESTSUse RPMDistribute full set of manifests/modules to each machineApply only the manifest relevant to that machine
PACKING PUPPET MANIFESTS IN RPMBuild an RPM containing everything under /etc/puppetMake files readable only by root
APPLY PUPPET MANIFESTSHave an RPM %postinst command apply the Puppet config
This isn't as straightforward as running the puppet apply from %postinstPuppet needs to install packages via yum, but yum is running installing the Puppet packageInstead, we work around with a dirty hack: have the %postinst create an at script whichchecks if yum has finished and then runs the puppet apply
RPM INSTALLATION AND MANAGEMENTHow do we get these RPMs on our machines?
PULPWe were already using PulpProvides yum repository managementUsed for managing security updates and deploying application code
http://pulpproject.org/
WHAT IS PULPRepository managerAllows us to easily audit what packages and versions are installed whereAllows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easy replication and clustering
HOW WE USE PULPPuppet contains details of what packages should be installedPulp manages which version of the package should be installedPulp allows us to clone repos and copy packages between them for easy qa>stage>live environmentmanagement
DEPLOYING CONFIGURATION AS CODEAllows us to reuse our existing code deployment infrastructureManage configuration deployment from Jenkins
HOW WE DEPLOY CODEEverything managed via the Jenkins continuous integration serverJenkins uses Pulp to install code on remote machines
DETAILS ON HOW WE DEPLOY CODEJenkins fetches code from source control (git)An RPM is builtTests are runIf tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)
DEPLOYMENT LIFECYCLEJenkins also manages deployment lifecycleRPMs are installed on stagingPromoted Builds plugin then used to install the same RPMs on live once testing is complete
PUPPET DEPLOYMENT PROCESSPuppet manifests are checked into gitLint tests via Jenkins pulls in modules with librarianpuppet, then builds an RPMDeployment to test environments, functional tests for wider codebase run
Jenkins Warnings plugin
PUTTING IT INTO PRODUCTIONOnce suitable tests (automated and manual) have been carried out, we promote Puppet config intoproductionWe use the Jenkins 'Promoted Builds' plugin for this
JENKINS: PROMOTION
EXCEPT..How does a machine get from a bare image to the state where we can push packages to it from Pulp?How does a machine know what type of machine it is?How do we find other resources, eg database hostname?
CLOUDFORMATIONAmazon tool for specifying infrastructureEverything* we provision inside AWS is provisioned via CloudFormationJSON templates
* Everything except for the things Amazon doesn't expose via CloudFormation..
CLOUDINITWorks with multiple cloud typesSorts out things like SSH keys, allows us to configure host namesAlso allows us to provide a bash script to run on startup
PROVISIONING A BARE INSTANCEcloudinit automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppetmanifests/modulesInstalls appropriate RPMsPuppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usualPuppet way
HOW DOES IT KNOW WHAT TYPE OF MACHINE IT IS?We tell it!Use an environmental variable $HOSTTYPESimply run
puppet apply \--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp
EXTRA FACTSCustom facter factsAlso specified in an environmental variable
Data comes from within the CloudFormation templateOn our list of things to look at:
FACTER_HOSTENVIRONMENT=liveFACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hieracloudformation
OTHER RESOURCESWe either:
Provide details as a facter factF̀ACTER_DBHOST=xyz
Also use this approach to limit distribution of secure details, eg DB passwords
Discover via the EC2 APIEg Varnish servers discover web backends by calling API and finding hosts taggedappropriately
FREE WINS!
FREE WINS!Greater control over the timing of Puppet runsImproved visibility for ops and devsConfiguration changes now have to be deployed to testing/staging first
MORE FREE WINS!Puppet configs now have a versionEasy to find config version on the machine itselfConfig changelogs accessible on every machine
(Git changelog added to RPM)
THE DOWNSIDESPuppet manifests and modules on all machines
Potentially a security issue?Mitigated by CloudFormation holding most sensitive data
ALTERNATIVE IMPLEMENTATIONSDon't want to use Pulp?Could do basically the same thing with yum s3 plugin Use mcollective to push package updates
https://github.com/jbraeuer/yums3plugin
FUTURE IMPROVEMENTSBuild AMIs using Packer instead of configuring at boot time
Decrease time to autoscaleWould probably still need to run Puppet at first boot to configure machine specific settings
QUESTIONS? COMMENTS?Sam Bashton
Twitter:
(Psst.. )
@bashtoni
http://www.bashton.com/jobs/