cobbler, func and puppet: tools for large scale environments

Cobbler, Func and Puppet:Cobbler, Func and Puppet:Tools for Large Scale EnvironmentsTools for Large Scale Environments

Francesco CrippaFrancesco Crippafcrippa @ fedoraproject,orgfcrippa @ fedoraproject,org

This presentation is licensed under a Creative CommonsAttribution-NonCommercial-ShareAlike (BY-NC-SA) 3.0 license.

what is a“Large Scale Environment”?

From this...

Google first WebFarm

...to this!

CC by Johnnie Walker: http://flickr.com/photos/johnniewalker/359440369

Large scale environments

● Huge number of machines● Huge number of environments

– Production

– Pre-production

– Test

– Develop

● Huge number of people● Need to scale!!!

All is complex

● An example: make a web server on line● In theory it should be easy:

– Buy hardware

– Install operating system (next, next, next...)

– Install a http server (yum install httpd)

– Configure http (vi /etc/httpd/conf/httpd.conf)

– That's all

● ...but...●

All is complex

● In a LSE you need:– 2 web server to maintain service in high availability

– 2 geographical sites to support disaster recovery

– A testing environment to test configurations

– A development environment to setup machines

– Backup

– Monitoring

– Clustering● Shared storage (SAN) required for each cluster

A lot of web servers

1... ...7 servers!

Where is the complexity?

● Web server configuration is simple● But you need to configure:

– 7 different web server (with exactly the same configuration!!!)

– Keep all configurations in sync

– 3 clusters (with different configurations)

– Shared storages (maybe with shared filesystems)

● The complexity is not on the “business”, but on the “infrastructure”

Typical work flow

● Installing OS to a new hardware● Configure machines as you need● Make (sometimes) unconventional tasks

InstallingInstallingO.S.O.S. ConfigurationsConfigurations UnconventionalUnconventional

TasksTasks

Typical work flow

InstallingInstallingO.S.O.S. ConfigurationsConfigurations UnconventionalUnconventional

TasksTasks

COBBLERCOBBLER PUPPETPUPPET FUNCFUNC

All under your control!

Copyright by WarnerBras (Matrix)

Installing OS...COBBLER

OS Installation

● Usually KickStart driven● Two main targets for a new installation:

– Real hardware

– Virtual Machine

● Usually expensive because require some manual human tasks– Put booting cd-rom into reader, configure network

and location of kickstart...

Cobbler

● Cobbler is a Linux provisioning server that allows for rapid setup of network installation environments.

● It keeps in sync:– DHCP

– TFTP

– RPM Repositories

– KickStarts

Importing repositories

● Check configuration files● Import repositories

[root@a~]# cobbler check[root@a~]# cobbler import 

­­mirror=rsync://servergoeshere/path/to/distro­­name=fedora9

Add profiles and systems

● Distros● Profiles● Systems

[root@a~]# cobbler distro add[root@a~]# cobbler profile add[root@a~]# cobbler system add

Keep in sync

● Generate output in:– /tftpboot

– /var/www/cobbler

– /etc/dhcp.conf

[root@a~]# cobbler sync

KickStart templates

● Kickstart is a template● You can pass values for all your variables

through “cobbler system add”

[root@a~]# cobbler system add ­­ksmeta=”KEY=us”

...keyboard $KEY...

Koan

● Client side● Provisioning for virtual machines ●

● koan --server=server.expl.org --profile=RHEL5-i386

●

● koan -–server=server.expl.org --profile=RHEL5-i386 --virt

Configure all your systemsPUPPET

Configuration Management

– Centralize all configurations into a single place

– Defining “templates” to generalize similar configurations (make one, apply hundreds)

– Manage exceptions

– Make possible for a lot of people to work all together

Working all together

Working all together

Working all together

Working all together

Configuration Items

– Files

– Packages

– Services

– Users

– Groups

– Mount points

– ...

– You can define your own items :-)

– Subscribe

– Require

– Notify

– ...

– Arrays

– Conditions

– ...

A formal definition for HTTPDClass http inherits security-hardening {

file {“/etc/httpd/httpd.conf”:

owner => apache,mode => 644,require => Package[“httpd”]

}

package {“httpd”:

ensure => “latest”}

service {“httpd”:

ensure => “running”,enabled => “true”,subscribe => [

File[“/etc/httpd/httpd.conf”],Package[“httpd”]

]}

}

A formal definition for HTTPD

node 'www.example.com' inherits default {import httpd

}

node 'default' {import baculaimport nagiosimport bacula-backup

}

Puppet architecture

– Client (the puppet) polls the server (the puppet-master) to know if there are new configuration files to download (every 30 minutes by default)

– Client is recognized through its hostname

– Server try to find a right configuration for that hostname, else it uses a “default” config.

– ...caching, md5, checksum and ssl are part of infrastructure

●

Python API for your networkFUNC

Goals

● Manage your hosts as python objects– Groups, ACLs, SSL certificates, ...

[root@a~]# func “*.com” call service restart “httpd”[root@a~]# func “*.lan” call command run “df ­h”

>>> from func.overlord import client>>> client1 = client.Client(“*.lan”)>>> client1.service.restart(“httpd”)>>> client1.command.run(“df ­h”)

Modules and new features

● Func based on modules architecture● A module support new stuff● 20 modules (libVirt, jboss, info, process,

command, iptables, nagios, etc)● Writing a new module is simple. ● When you write a module, it works on both CLI

and PyScripting, no modification on func are needed.

THANK YOU.

Credits: Byte-Code, Tango Icons, Crystals Icons