scale12x build a cloud day: chef: the swiss army knife of cloud infrastructure
DESCRIPTION
Chef is an open source configuration management and automation framework used to configure, deploy and manage infrastructure of every type. Deploying to the cloud has made it easy to run large numbers of servers and Chef makes it even easier to deploy to nearly every public and private cloud platform as well as virtualized and physical servers. This talk will provide a quick introduction to Chef and is intended for sysadmins and developers familiar with the concepts behind managing applications and infrastructure in the cloud, without diving too deeply into technical specifics.TRANSCRIPT
![Page 1: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/1.jpg)
![Page 2: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/2.jpg)
Chef: The Swiss Army Knife of Cloud InfrastructureMatt Ray SCALE Build a Cloud Day February 21, 2014
![Page 3: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/3.jpg)
Introductions• Matt Ray
• Director of Cloud Integrations at Chef
• mattray GitHub|IRC|Twitter
![Page 4: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/4.jpg)
We have a problem…
![Page 5: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/5.jpg)
http://www.flickr.com/photos/michaelheiss/3090102907/
Complexity
![Page 6: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/6.jpg)
Items of Manipulation (Resources)• Nodes • Networking • Files • Directories • Symlinks • Mounts
• Routes • Users • Groups • Packages • Services • Filesystems
![Page 7: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/7.jpg)
Application
A tale of growth...
![Page 8: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/8.jpg)
Application
Application Database
Add a database
![Page 9: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/9.jpg)
Application
App Databases
Make database redundant
![Page 10: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/10.jpg)
App Servers
App Databases
Application server redundancy
![Page 11: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/11.jpg)
App LB
App Servers
App Databases
Add a load balancer
![Page 12: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/12.jpg)
App LBs
App Servers
App Databases
Webscale!
![Page 13: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/13.jpg)
App LBs
App Servers
App DB Cache
App DBs
Now we need a caching layer
![Page 14: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/14.jpg)
App LBs
App Servers
App DB Cache
App DBs
Infrastructure has a Topology
![Page 15: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/15.jpg)
Round Robin DNS
App Servers
App DB Cache
App DBs
Floating IP?
Your Infrastructure is a Snowflake
![Page 16: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/16.jpg)
App LBs
App Servers
< Shiny!
DB slaves
Cache
DB Cache
DBs
Complexity Increases Quickly
Are we monitoring??
![Page 17: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/17.jpg)
™
![Page 18: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/18.jpg)
The Chef Framework• Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults
http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/
![Page 19: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/19.jpg)
The Chef Tool(s)• ohai • chef-client • chef-shell • knife • The Ruby language
Omnibus - Full Stack Native Packages
![Page 20: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/20.jpg)
The Chef API• HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services?
http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
![Page 21: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/21.jpg)
The Chef Community• Apache License, Version 2.0 • Thousands of Individual and Corporate contributors. • Thousands of cookbooks available from the
community • http://community.opscode.com
![Page 22: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/22.jpg)
How does it work?
http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg
![Page 23: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/23.jpg)
Chef is Infrastructure as Code• Programmatically
provision and configure
• Treat like any other code base
• Reconstruct business from code repository, data backup, and bare metal resources.http://www.flickr.com/photos/louisb/4555295187/
![Page 24: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/24.jpg)
Programs• Chef generates
configurations directly on nodes from their run list
• Reduce management complexity through abstraction
• Store the configuration of your programs in version control
http://www.flickr.com/photos/ssoosay/5126146763/
![Page 25: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/25.jpg)
Declarative Interface to Resources• Define Policy • Say what, not how • Pull not Push
http://www.flickr.com/photos/bixentro/2591838509/
![Page 26: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/26.jpg)
That Looks Like Thispackage "apache2"
template "/etc/apache2/apache2.conf" do! source "apache2.conf.erb"! owner "root"! group "root"! mode "0644"! variables(:allow_override => "All")! notifies :reload, "service[apache2]"!end
service "apache2" do! action [:enable,:start]! supports :reload => true!end
![Page 27: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/27.jpg)
http://www.flickr.com/photos/peterrosbjerg/3913766224/
Chef Nodes• Chef runs on nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search
![Page 28: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/28.jpg)
Search• Search for nodes with Roles • Find Topology Data !
• IP addresses • Hostnames • FQDNs
http://www.flickr.com/photos/kathycsus/2686772625
![Page 29: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/29.jpg)
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
So when this...
![Page 30: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/30.jpg)
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
...becomes this
![Page 31: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/31.jpg)
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
!!! !!
Jboss App
Memcache
Postgres Slaves
Postgres Master
NagiosGraphite
...this can happen automatically
![Page 32: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/32.jpg)
NagiosGraphite
!!! !!
NagiosGraphite
Memcache
Postgres Slaves
• Load balancer config
• Nagios host ping
• Nagios host ssh
• Nagios host HTTP
• Nagios host app health
• Graphite CPU
• Graphite Memory
• Graphite Disk
• Graphite SNMP
• Memcache firewall
• Postgres firewall
• Postgres authZ config
•12+ resource changes for 1 node addition
Count the Resources
Jboss App
![Page 33: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/33.jpg)
Landscape of Chef-managed Infrastructure
![Page 34: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/34.jpg)
knife
![Page 35: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/35.jpg)
knife - with the Chef Server• knife node
• create / edit / delete • list
• knife cookbook ... • knife role ... • knife environment ...
![Page 36: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/36.jpg)
• SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
$ knife bootstrap SERVER -r ‘role[webserver]’
knife bootstrap
![Page 37: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/37.jpg)
gem install knife-cloudstack
Chef & CloudStack
![Page 38: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/38.jpg)
• ** CS COMMANDS ** • knife cs aag list (options) • knife cs account list (options) • knife cs cluster list (options) • knife cs config list (options) • knife cs disk list (options) • knife cs domain list (options) • knife cs firewallrule create hostname 8080:8090:TCP:10.0.0.0/24 • knife cs firewallrule list (options) • knife cs forwardrule create hostname 8080:8090:TCP • knife cs host list (options)
knife cs
knife-cloudstack 1/4
![Page 39: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/39.jpg)
• knife cs hosts • knife cs iso list (options) • knife cs keypair create KEY_NAME (options) • knife cs keypair delete KEY_NAME (options) • knife cs keypair list (options) • knife cs network list (options) • knife cs oscategory list (options) • knife cs ostype list (options) • knife cs pod list (options) • knife cs project list (options)
knife cs
knife-cloudstack 2/4
![Page 40: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/40.jpg)
• knife cs publicip list (options) • knife cs router list (options) • knife cs securitygroup list (options) • knife cs server add nic SERVERID NETWORKID (--ipaddress X.X.X.X) • knife cs server create [SERVER_NAME] (options) • knife cs server delete SERVER_NAME [SERVER_NAME ...] (options) • knife cs server list (options) • knife cs server reboot SERVER_NAME [SERVER_NAME ...] (options) • knife cs server remove nic SERVERID NICID • knife cs server start SERVER_NAME [SERVER_NAME ...] (options)
knife cs
knife-cloudstack 3/4
![Page 41: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/41.jpg)
• knife cs server stop SERVER_NAME [SERVER_NAME ...] (options) • knife cs service list (options) • knife cs stack create JSON_FILE (options) • knife cs stack delete JSON_FILE (options) • knife cs template create NAME (options) • knife cs template extract NAME (options) • knife cs template list (options) • knife cs template register NAME (options) • knife cs user list (options) • knife cs volume create NAME (options) • knife cs volume list (options) • knife cs zone list (options)
knife cs
knife-cloudstack 4/4
![Page 42: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/42.jpg)
• Instant infrastructure • Unlimited capacity • Autoscaling • No commitment • Immediate replacement
Why the Cloud?
![Page 43: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/43.jpg)
• Reliability • Performance • Security • Price
Why not the Cloud?
![Page 44: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/44.jpg)
Data Gravity
![Page 45: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/45.jpg)
Know our escape plan for every infrastructure provider
![Page 46: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/46.jpg)
Chef for Infrastructure Portability
• knife rackspace
• knife openstack
• knife azure
• knife cloudstack
• knife ec2
• knife google
• knife hp
• knife vsphere
• ...and many others
![Page 47: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/47.jpg)
• AWS
• Rackspace
• HP
• Azure
• many others
™
Desktop, Virtualization & Cloud
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
![Page 48: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/48.jpg)
• AWS
• Rackspace
• HP
• Azure
• many others
Desktop, Virtualization & Cloud
• Vagrant
• VMware
• CloudStack
• Eucalyptus
• OpenStack
• bare metal
™
![Page 49: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/49.jpg)
What does this all mean?•Every infrastructure is a unique snowflake •Understand the costs associated with the features of your platform(s) of choice. •Chef enables Infrastructure Portability •Use the same infrastructure code for wherever you deploy •"Data Gravity" is a concern
™
![Page 50: SCALE12X Build a Cloud Day: Chef: The Swiss Army Knife of Cloud Infrastructure](https://reader038.vdocuments.mx/reader038/viewer/2022110115/54c688cc4a7959fb258b45de/html5/thumbnails/50.jpg)
There’s a lot more...• Attributes • Environments • Roles • Chef Handlers • LWRPs • Workflow Helpers • Testing Frameworks • and even more awesome