Puppet at the centre of everything

David Mytton

Woop Japan!

Server Density Architecture

•100 servers - Ubuntu 12.04

Server Density Architecture

•100 servers - Ubuntu 12.04

•50% virtual

Server Density Architecture

•100 servers - Ubuntu 12.04

•Nginx, Python, MongoDB

•50% virtual

Server Density Architecture

•100 servers - Ubuntu 12.04

•Nginx, Python, MongoDB

•25TB data per month

•50% virtual

Use cases

• Config

Use cases

• Config

• Failover

Use cases

• Config

• Code deploys

• Failover

Use cases

• Config

• Code deploys

• System updates

• Failover

• Base manifest

Config

• Base manifest

Config

• Sub manifests

• base

• softlayer

• softlayer-dedicated

• rackspace

• ...

Failover

• /etc/hosts

<%= metrics2IP %> alerts.honshuu.prod audit.honshuu.prod auth.honshuu.prod cloud.honshuu.prod datastore.honshuu.prod inventory.honshuu.prod ...

Failover

• /etc/hosts

108.168.254.33 puppet

Expected

Dealing with outages

Really test your vendors

Dealing with outages

Simulations

Dealing with outages

• Commit

Code deploys

• Commit

Code deploys

• Build

• Commit

Code deploys

• Build

• Deploy

/opt/puppet/bin/mco puppetd runonce -I <host>

• Commit

Code deploys

• Build

• Deploy

• Puppet run

Code deploys

Advantages

APIs

Code deploys

Advantages

APIs

Already there

Code deploys

Advantages

APIs

Already there

System facts

Code deploys

Disadvantages

Slow

Code deploys

Disadvantages

Slow

Puppet run already

Code deploys

Disadvantages

Slow

Puppet run already

Eventually consistent

System updates

• Bash script

System updates

• Bash script

• Remote commands

System updates

• Bash script

• Remote commands

• Notifications

System updates

• Bash script

• Remote commands

• Notifications

• Canaries

•Build master (buildbot): VM x2 CPU 2.0Ghz, 2GB RAM – $89/m

•Build slave (buildbot): VM x1 CPU 2.0Ghz, 1GB RAM – $40/m

•Staging load balancer: VM x1 CPU 2.0Ghz, 1GB RAM – $40/m

•Staging server 1: VM x2 CPU 2.0Ghz, 8GB RAM – $165/m

•Staging server 2: VM x1 CPU 2.0Ghz, 2GB RAM – $50/m

•Puppet master: VM x2 CPU 2.0Ghz, 2GB RAM – $89/m

Total: $473/m

Colo experiment

Colo experiment

•Dell 1U R415

•x2 8C AMD 2.8Ghz

•32GB RAM

Colo experiment

•Dell 1U R415

•x2 8C AMD 2.8Ghz

•32GB RAM

•Dual PSU, NIC

Colo experiment

•Dell 1U R415

•x2 8C AMD 2.8Ghz

•32GB RAM

•Dual PSU, NIC

•x4 1TB SATA hot swappable

Dedicated?

Dedicated?

•Hardware replacement

Dedicated?

•Hardware replacement

•Managed/support

Dedicated?

•Hardware replacement

•Managed/support

•Networking

Colo?

Colo?

•Hardware spec/value

Colo?

•Hardware spec/value

•Total cost

Colo?

•Hardware spec/value

•Total cost

•Internal skills?

Colo?

•Hardware spec/value

•Total cost

•Internal skills?

•More fun?!

Colo: Power

•£300-350/kWh/m

•4.5A = £520/m

•9A = £900/m

David Mytton

david@serverdensity.com

@davidmytton

Woop Japan!

blog.serverdensity.com