infrastructure as code with chef / puppet

Infrastructure as Code with Chef / Puppet

Edmund Haselwanter ([email protected])

Samstag, 16. November 13

mailto:[email protected]


cloudbau

Edmund Haselwanter

●Co-Founder and CSO cloudbau GmbH

●Dell Professional Services (Openstack/Crowbar/Automation)

●So!ware Development Background

●10 Years Infrastructure Automation

●@ehaselwanter on twitter/github/irc/...

Introduction


cloudbau

Business-Agility needs IT-Agility

„IT REVOLUTION MANIFESTO“, http://itrevolution.com

For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change.

Gene Kim, Kevin Behr, George Spafford, 2013

E.M. Goldratt, 1984

Theory of Constraints → →


http://itrevolution.com/

http://itrevolution.com/

cloudbau

IT-Agility and the „Risk of Change“

The core, chronic con!ict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).

„Lowering risk of change through tools and culture“John Allspaw, Paul Hammond, 2009

small changes, o!en

reproducible

process standards

Request Approve

Test Deploy

Develop

Water-

Scrum

- FallGene Kim

expect failure

cooperative culture

metrics


cloudbau

Tools and Culture

CPU-Virt.

Storage-Virt.

SDN

Con!g-DB

Programmable Infrastructure Description in Code

Process and Culture

Continuos Delivery

DevOpsCross-Functional Teams

Automatic Build

Anti-Fragile Organizations

Binary-Repo DescriptionOrchestration

Prod Env Stage Env Dev Env


cloudbau

Con!guration Management


cloudbau

●Just build it

●Keep notes in server.txt

●Move notes to the wiki

●Custom scripts (in scm?!)

●Snapshot & Clone

Evolving towards Con!guration Management


cloudbau

Applications

http://www."ickr.com/photos/steffenz/337700069/

http://www."ickr.com/photos/kky/704056791/


http://www.flickr.com/photos/steffenz/337700069/








cloudbau

Infrastructure

http://www."ickr.com/photos/sbh/462754460/


http://www.flickr.com/photos/sbh/462754460/

http://www.flickr.com/photos/sbh/462754460/

cloudbau

Collection of Resources

http://www."ickr.com/photos/philliecasablanca/3354734116/

• Networking

• Files

• Directories

• Symlinks

• Mounts

• Routes

• Users

• Groups

• Tasks

• Packages

• Software

• Services

• Configuration

• Other Stuff


cloudbau

Acting in Concert

http://www."ickr.com/photos/glowjangles/4081048126/


http://www.flickr.com/photos/glowjangles/4081048126/

http://www.flickr.com/photos/glowjangles/4081048126/

cloudbau

To Provide a Service

http://www."ickr.com/photos/28309157@N08/3743455858/


http://www.flickr.com/photos/28309157@N08/3743455858/


cloudbau

And it Evolves

http://www."ickr.com/photos/16339684@N00/2681435235/




cloudbau

Application Server

See Node


cloudbau

Application Server

Application Database

See Nodes


cloudbau

Application Server

Application Databases

See Nodes Grow


cloudbau

Application Servers


See Nodes Grow


cloudbau

Application Servers


Load Balancer

See Nodes Grow


cloudbau

Application Servers


Load Balancers

See Nodes Grow


cloudbau

Application Servers

Application Database Cache

Load Balancers


See Nodes Grow


cloudbau

Application Servers

Application Database Cache

Load Balancers


Tied Together with Con!guration


cloudbau

So when this

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


cloudbau

Becomes This

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


cloudbau

This can happen automagically

Jboss App

Memcache

Postgres Slaves

Postgres Master

NagiosGraphite


cloudbau

Count the Resources

NagiosGraphite

Jboss App

Memcache

Postgres Slaves

• Load balancer config

• Nagios host ping

• Nagios host ssh

• Nagios host HTTP

• Nagios host app health

• Graphite CPU

• Graphite Memory

• Graphite Disk

• Graphite SNMP

• Memcache firewall

• Postgres firewall

• Postgres authZ config

• 12+ resource changes for 1 node addition


cloudbau

Focus Today: Chef and Puppet

Series A: 2,5 M$ (2009)Series B: 11 M$ (2010)

Series C: 19,5 M$ (2012)

Technologie

Series A: 2 M$ (2009)Series B: 5 M$ (2010)

Series C: 8,5 M$ (2011)Series D: 30 M$ (2013)

Technologie

*alle Angaben von www.crunchbase.com


cloudbau

Chef & Puppet

since 2005 since 2009

Declarative DSL Ruby DSL

Customers: Customers:

• CERN • AWS (OpsWorks)

• Twitter • Facebook

• Zynga • SAP

•VMware • IBM (Smartcloud)

• Microso#


cloudbau

What is Chef?

Recipes and Cookbooks that describe and deliver code.

Chef enables people to easily build & manage complex & dynamic applications at massive scale.

• New model for describing infrastructure that promotes reuse

•  Programmatically provision and configure • Reconstruct business from code repository,

data backup, and bare metal resources

Chef is an IT automation platform for developers & systems engineers to continuously define, build, and manage infrastructure.

CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!


cloudbau

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!


cloudbau


cloudbau

Puppet is IT automation so"ware that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and con$guration to orchestration and reporting.

What is Puppet?

How Puppet WorksPuppet uses a declarative, model-based approach to IT automation.

1. Define the desired state of the infrastructure's configuration using Puppet's declarative configuration language.

2. Simulate configuration changes before enforcing them.

3. Enforce the deployed desired state automatically, correcting any configuration drift.

4. Report on the differences between actual and desired states and any changes made enforcing the desired state.


cloudbau

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!

What is Chef?







CHEF USES:

“!”!


cloudbau


cloudbau

Concepts


cloudbau

Declarative Language

File 1

Package A

Service IPackage B

Template 1


cloudbau

Convergence to desired State

File 1

Package A

Service IPackage B

Template 1

File 1

Package A

“run”

State X State Y


cloudbau

Idempotence

File 1

Package A

Service IPackage B

Template 1

“run”

State Y State Y

File 1

Package A

Service IPackage B

Template 1


cloudbau

The Implementations

Puppet and Chef


cloudbau

Terminology

Java Puppet Chef

*.java manifests *.pp recipes *.rb

package module cookbook


cloudbau

Terminology

Java Puppet Chef

singleton parametrized classes

recipes with attributes

class de#nes, classes de#nitions, libraries


cloudbau

Syntax

Puppet Chef

Limited DSL (by design) DSL + full ruby

Extended by De!nitionsDe!nitions, Resource/Providers, Light

Weight Resource

Custom Resource providersDe!nitions, Resource/Providers, Light

Weight Resource

ERB & Custom Libraries Providers, Libraries


cloudbau

Syntax Examples

Puppet Chef

class myapache { package “apache2” service “apache2”: ensure => “running”, require => Package[“apache2”]}

package “apache2”

service “apache2 do action [:enable , :start]end


cloudbau

Variables

Puppet Chef

#special syntax

$bla = “aa”

#string interpolation

$bla_string = “${bla}”

#native ruby

bla = “aa”

#string interpolation

bla_string = #{bla}


cloudbau

Loops

Puppet Chef

pass array of elements

erb template for loops

create_resource (:type, hash)

do while , repeat etc. available through native ruby


cloudbau

Conditionals

Puppet (since 2.7) Chef

if

case (execute block)

selectors (get value)

if, unless, case etc. available through native ruby


cloudbau

Project Structure

Puppet Modules Chef

files/

lib/

README.rdoc

manifests/

templates/

attributes/

definitions/

files/

libraries/

metadata.rb

providers/

README.rdoc

recipes/

resources/

templates/


cloudbau

Execution Model

Puppet Chef

Compile catalog (directed graph model) Then Execute Top Down execution

only code for state transition is visited every line of code is visited all the time


cloudbau

Execution Model

Puppet Chef

standalone standalone

client - serverclient - server

hosted


cloudbau

System Information

Puppet/Factor => Flat Key/Value

"hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" },

Chef/Ohai => Nested Hash


cloudbau

●Selects Package Provider

●Selects Service Provider

●....

System Information


cloudbau

System Information (Chef Example)

execute "load sysctl" do command "/sbin/sysctl -p" action :nothingend

bytes = node['memory']['total'].split("kB")[0].to_i * 1024 / 3,pages = node['memory']['total'].split("kB")[0].to_i * 1024 / 3 / 2048

# adjust shared memory and semaphorestemplate "/etc/sysctl.conf" do source "sysctl.conf.erb" variables( :shmmax_in_bytes => bytes, :shmall_in_pages => pages ) notifies :run, "execute[load sysctl]", :immediatelyend


cloudbau

Templates

# Set up application listeners here.

listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%><% if node["haproxy"]["enable_admin"] -%>listen admin 0.0.0.0:22002 mode http stats uri /<% end -%>


cloudbau

Lint Tools

http://acrmp.github.io/foodcritic/

http://puppet-lint.com/




http://puppet-lint.com

http://puppet-lint.com

cloudbau

Dependency Managementhttps://github.com/applicationsonline/librarian-chef

http://librarian-puppet.com

http://berkshelf.com


https://github.com/applicationsonline/librarian-chef

https://github.com/applicationsonline/librarian-chef





cloudbau

Reports


cloudbau

Web Interface


cloudbau

●Razor for bare metal provisioning

●mcollectiv for remote execution/orchestration

●test-kitchen for infrastructure testing

●vagrant for rapid development

● ....

Tools


cloudbau

●Switch Con#g

●Driven by SDN Adoption

●Lots of Vendors (Arista ..)

Networking


cloudbau

Community


cloudbau

Chef Community

Community Overview!

25,000+ Community Members!!

1,000+ Community "Cookbooks!

!250,000+ Cookbook

Downloads!!

400+ Public Training Attendees in the last year!

!30+ Meetup Groups!

!

Over 200 Corporate Contributors


cloudbau

http://community.opscode.com/cookbooks

approx 1200 co0kbooks134 maintained by Opscode




cloudbau

http://forge.puppetlabs.com

approx 1700 modules72 maintained by Puppetlabs




cloudbau

The Price Tag


cloudbau

Puppet: How to BuyCumulative # of

NodesPer Node

Standard SupportPer Node Premium

Support*1-10 Download FREE NA

11-99 $ 99 Contact Sales

100-249 $ 93 Contact Sales

250-499 $ 88 $ 152

500-999 $ 83 $ 119

1000-2499 $ 79 $ 99

2500+ Contact Sales Contact Saleshttps://puppetlabs.com/puppet/how-to-buy


http://info.puppetlabs.com/download-pe.html

http://info.puppetlabs.com/download-pe.html









https://puppetlabs.com/puppet/how-to-buy

https://puppetlabs.com/puppet/how-to-buy

cloudbau

Puppet Enterprise vs. OpenSource


cloudbau

Chef: How to Buy

Free Launch Standard Premium

Price per Month Free $ 120 $ 300 $ 600

Nodes 5 20 50 100

Standard Support — ✔ ✔ ✔

http://www.opscode.com/enterprise-chef/#plans-pricing




cloudbau

Enterprise Chef vs. OpenSource Chef


Danke!

Edmund Haselwanter

@[email protected]+49 30 57701800www.cloudbau.de

cloudbau GmbHKörnerstr. 7-1010785 Berlin

cloudbau









Credits:• Patrick Debois (http://www.jedi.be/blog/) for some of his Puppet/Chef Slides• Opscode (http://www.opscode.com) for product info and some slides • Puppetlabs (http://puppetlabs.com) for product info


http://www.jedi.be/blog/

http://www.jedi.be/blog/

http://www.opscode.com




infrastructure as code with chef / puppet

Technology

culturecloudbau samstag

technologiecloudbau

concertcloudbau samstag

servicecloudbau samstag

evolvescloudbau samstag

infrastructurecloudbau

postgres master

series c