puppet camp silicon valley 2015: how tubemogul reached 10,000 puppet deployment in one year
TRANSCRIPT
How TubeMogul reached 10,000 Puppet deployment in
one yearMay 26th, 2015
Nicolas Brousse | Sr. Director Of Operations Engineering | [email protected]
Julien Fabre | Site Reliability Engineer | [email protected]
Who are we?
TubeMogul● Enterprise software company for digital branding● Over 27 Billions Ads served in 2014● Over 30 Billions Ad Auctions per day● Bid processed in less than 50 ms● Bid served in less than 80 ms (include network round trip)● 5 PB of monthly video traffic served● 1.3 EB of data stored
Who are we?
Operations Engineering● Ensure the smooth day to day operation of the platform
infrastructure● Provide a cost effective and cutting edge infrastructure● Team composed of SREs, SEs and DBAs● Managing over 2,500 servers (virtual and physical)
Our Infrastructure
Public Cloud On Premises
Multiple locations with a mix of Public Cloud and On Premises
● Java (a lot!)● MySQL● Couchbase● Vertica● Kafka● Storm● Zookeeper, Exhibitor● Hadoop, HBase, Hive● Terracotta● ElasticSearch, Kibana● LogStash● PHP, Python, Ruby, Go...● Apache httpd● Nagios● Ganglia
Technology Hoarders
● Graphite● Memcached● Puppet● HAproxy● OpenStack● Git and Gerrit● Gor● ActiveMQ● OpenLDAP● Redis● Blackbox● Jenkins, Sonar● Tomcat● Jetty (embedded)● AWS DynamoDB, EC2, S3...
● 2008 - 2010: Use SVN, Bash scripts and custom templates.
● 2010: Managing about 250 instances. Start looking at Puppet.
● 2011: Puppet 0.25 then 2.7 by EOY on 400 servers with 2 contributors.
● 2012: 800 servers managed by Puppet. 4 contributors.
● 2013: 1,000 servers managed by Puppet. 6 contributors.
● 2014: 1,500 servers managed by Puppet. Introduced Continuous Delivery Workflow. 9 contributors. Start 3.7 migration.
● 2015: 2,000 servers managed by Puppet. 13 contributors.
Five Years Of Puppet!
● 2000 nodes
● 225 unique nodes definition
● 1 puppetmaster
● 112 Puppet modules
Puppet Stats
● Virtual and Physical Servers Configuration : Master mode
● Building AWS AMI with Packer : Master mode
● Local development environment with Vagrant : Master mode
● OpenStack deployment : Masterless mode
Where and how do we use Puppet ?
Code Review?
● Gerrit, an industry standard : Eclipse, Google, Chromium, OpenStack, WikiMedia, LibreOffice, Spotify, GlusterFS, etc...
● Fine Grained Permissions Rules● Plugged to LDAP● Code Review per commit● Stream Events● Use GitBlit● Integrated with Jenkins and Jira● Managing about 600 Git repositories
A Powerful Gerrit Integration
Gerrit in Action
verify -1 when no ticket # or doesn’t pass Jenkins code validation
● 1 job per module● 1 job for the manifests and hiera data● 1 job for the Puppet fileserver● 1 job to deploy
Continuous Delivery with Jenkins
Global Jenkins stats for the past year● ~10,000 Puppet deployment● Over 8,500 Production App Deployment
Plugin : github.com/jenkinsci/job-dsl-plugin
● Automate the jobs creation
● Ensure a standard across all the jobs
● Versioned the configuration
● Apply changes to all your jobs without pain
● Test your configuration changes
Jenkins job DSL : code your Jenkins jobs
Team Awareness: HipChat Integration with Hubot
Infrastructure As Code● Follow standard development lifecycle● Repeatable and consistent server
provisioning
Continuous Delivery● Iterate quickly● Automated code review to improve code
quality
Reliability● Improve Production Stability● Enforce Better Security Practices
Puppet Continuous Delivery Workflow: The Vision
The Workflow
The Workflow : Puppet code logic
Puppet environments● Dedicated node manifests (*.pp)● Modules deployed by branch with Git submodules
All the data in Hiera● Try to avoid params.pp class● Store everything : modules parameters, classes, keys, passwords, ...
Puppet Code Hierarchy
/etc/puppet├── puppet.conf, hiera.yaml, *.conf├── hiera└── environments ├── dev │ ├── manifests │ │ ├── nodes/*.pp │ │ └── site.pp │ └── modules │ ├── activemq │ ... │ └── zookeeper └── production ├── manifests │ ├── nodes/*.pp │ └── site.pp └── modules ├── activemq … └── zookeeper
Git submodules, branch dev
Git submodules, branch production
Hiera Configuration
$ cat /etc/puppet/hiera.yaml---:backends: - eyaml - yaml:yaml: :datadir: /etc/puppet/hiera:eyaml: :pkcs7_private_key: /var/lib/puppet/hiera_keys/private_key.pkcs7.pem :pkcs7_public_key: /var/lib/puppet/hiera_keys/public_key.pkcs7.pem:hierarchy: - fqdn/%{::fqdn} - "%{::zone}/%{::vpc}/%{::hostgroup}" - "%{::zone}/%{::vpc}/all" - "%{::zone}/%{::hostgroup}" - "%{::zone}/all" - hostname/%{::hostname} - hostgroup/%{::hostgroup} - environment/%{::environment} - common:merge_behavior: deeper
Hiera eyaml : github.com/TomPoulton/hiera-eyaml
● Hiera backend● Easy to use● Powerful CLI : eyaml edit /etc/puppet/hiera/secrets.yaml
Encrypt Your Secrets
$ cat secret.yaml---ec2::access_key_id: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAVIa28OwyaqI5N1TDCvVkBZz3YG+s+Hfzr0lqgcvRCIuJGpq28sQmmuBaQjWY38i86ZSFu0gM6saOHfG64OzVlurO7k/l0CKeL0JfXNaVM4TUqMaN9dSkL5e2vsmpLKrMASawmarqbLYwllTrTe32H4NWxU1e+qWLeUMr9ciBnA3W1Azm4RIo+3bsvgvMfdks....=]
Encrypt Files
Blackbox : github.com/StackExchange/blackbox
● Use GPG to encrypt secret files● Easy to add/delete team members● No need to change your Puppet code !
# modules/${modules_name}/files/credentials.yaml.gpg
file { ‘/etc/app/credentials.yaml’: ensure => ‘file’, owner => ‘root’, group => ‘root’, mode => ‘0644’, source => ‘puppet:///modules/${module_name}/credentials.yaml’}
The Workflow
The Workflow : bottlenecks
● Only Ops team members can commit (SRE, SE)
● Review and validation is done only by a SRE
● Jenkins will verify the code but will not validate the commit
● Static Puppet environments
● Rely a lot on server hostnames
Flexibility : R10K github.com/adrienthebo/r10k !
● Dynamic environments
● No Git submodules anymore ! : - )
● Easy to reproduce any environment
● Can use private and forge Puppet modules
● Can use branches and tags
● Based on Puppetfile
Puppet Workflow Reloaded!
R10K
$ cat Puppetfileforge "https://forgeapi.puppetlabs.com"
# Forge modulesmod 'pdxcat/collectd'mod 'puppetlabs/rabbitmq'mod 'arioch/redis'mod 'maestrodev/wget'mod 'puppetlabs/apt'mod 'puppetlabs/stdlib'
# Tubemogul modulesmod "hosts", :git => 'ssh://<gerrit_host>/puppet/modules/hosts', :branch => 'dev'mod "timezone", :git => 'ssh://<gerrit_host>/puppet/modules/timezone', :branch => 'dev'
...
Puppet Workflow Reloaded!
Better code organization : Roles and Profiles
● Represent the business logic : Roleso Highest abstraction layero Use Profiles for implementation
● Implement the applications : Profileso Remove potential code duplicationo Use modules and other Puppet resources
Roles/Profiles Pattern
class role::logs { include profile::base include profile::logstash::server include profile::elasticsearch}
class profile::logstash { $version = hiera('profile::logstash::server::version', '1.4.2') $es_host = hiera('profile::logstash::server::es_host', 'es01') $redis_host = hiera('profile::logstash::server::redis_host', 'redis01')
class { 'logstash': package_url => "https://download.elasticsearch.org/logstash/.../logstash_${version}.deb", java_install => true, }
logstash::configfile { 'input_redis': content => template('logstash/configfile/logstash.input_redis.conf.erb'), order => 10, }
logstash::configfile { 'output_es': content => template('logstash/configfile/logstash.output_es.conf.erb'), order => 30, }}
Do not rely on hostname : nodeless approach
● Facts to guide Puppet● No node myawesomeserver { } anymore● Enforce a cluster vision● site.pp gives the configuration logic
Puppet Workflow Reloaded!
# /etc/puppet/manifests/site.pp
node default {
if $::ec2_tag_tm_role { notify { "Using role : ${ec2_tag_tm_role}": } include "role::${::ec2_tag_tm_role}" } else { fail(‘No role found. Nothing to configure.’) }
}
● Specify tags during the provisioning
● Retrieve tags with AWS Ruby SDK and create facts
● New hierarchy
AWS EC2 tags
$ facter -p | grep ec2_tagec2_tag_cluster => rtb-bidderec2_tag_nagios_host => mgmt01ec2_tag_name => bidderec2_tag_pupenv => productionec2_tag_tm_role => rtb::bidder
:hierarchy: - "%{::zone}/%{::ec2_tag_vpc}/%{::ec2_tag_cluster}" - "%{::zone}/%{::ec2_tag_vpc}/all" - "%{::zone}/all" - vpc/%{::ec2_tag_vpc}/%{::ec2_tag_cluster} - vpc/%{::ec2_tag_vpc}/all - environment/%{::environment} - common
New merging and reviewing rules
● Everyone can commit a Puppet code
● Allow everyone to review a Puppet change (+1)
● Allow SE and SRE to validate a Puppet change (+2)
● Auto validation/merging in dev if at least 80% of test (+2)
Next improvements
● Acceptance testing with Beaker and Docker
● Full test provisioning with ServerSpec
● PuppetDB to improve the reporting
● Dedicated Puppet Masters
Nicolas BrousseJulien Fabre
@orieg@julien_fabre