continuous integration for the world - xpdays benelux 2009
DESCRIPTION
Infrastructure is becoming code. In this presentation we draw the parallels between the automation of systems provisioning and development. Continuous integration is a good candidate to have your developers and sysadmins give each other faster feedback on changes and their impact on both sides. This was presented by Patrick Debois @xpdays Benelux 2009. With many thanks to Gildas Le Nadan who worked together on the first version @xpdays France 2009TRANSCRIPT
Patrick Debois
Continuous Integration for
the world
Wednesday 25 November 2009
Patrick Debois
Wednesday 25 November 2009
My name is Patrick Debois and I come from Belgium
Wednesday 25 November 2009
Iʼm a freelancer always on the look out for new interesting things
Wednesday 25 November 2009
I have a sysadmin background
Wednesday 25 November 2009
Now for the past few years, Iʼve been looking together with other sysadmins with great interest to the Agile Developers community at what we could learn from Agile development
Wednesday 25 November 2009
And this year I wanted to give you my own version of Continuous Integration
Continuous Integration is the practice of integrating early and often, so as to avoid the pitfalls of "integration hell". The ultimate goal is to reduce timely rework and thus reduce cost and time. When done well, continuous integration has been shown to achieve these goals.
Wednesday 25 November 2009
Let’s start with the definition on wikipedia.
http://www.martinfowler.com/articles/continuousIntegration.html
http://en.wikipedia.org/wiki/Continuous_Integration
Traditional CI Overview
http://www.javaworld.com/javaworld/jw-12-2008/images/CIOverview.jpgWednesday 25 November 2009
Different parts: Source Control, Build, Test, Package, Deploy/Publish
Important to remember
Traditional CI Flow
Courtesy Chris Read / ThoughtWorksWednesday 25 November 2009
Courtesy Chris Read / ThoughtWorkshttp://www.slideshare.net/ChristopherRead/continuous-integration-build-pipelines-and-continuous-deployment
CI Maturity Model
http://www.anthillpro.com/blogs/anthillpro-blog/2009/05/05/1241542860000.htmlWednesday 25 November 2009
Yes some call it insane.
Build Pipeline Flow
Courtesy Chris Read / ThoughtWorksWednesday 25 November 2009
http://www.slideshare.net/ChristopherRead/continuous-integration-build-pipelines-and-continuous-deployment
the flow - it it passes from Integration to UATthen to the next environment
Huge problem is managing all these environments and keeping them aligned
Definition of Done
Wednesday 25 November 2009
We need to go beyond the traditional testing and extend it to the live system
Agility in Production• redefine key concept
• « the software works »
• « the software works in production »
• = beyond silos
Wednesday 25 November 2009
http://www.lifeinamatrix.com/wp-content/uploads/2009/01/life-in-a-matrix-9-breaking-the-silos.jpg
Who is part of the team
• developers
• testers
• analysts
• project managers
• usability experts
• architects
Wednesday 25 November 2009
But usually we forget
• the sysadmins
• the guys from operations
Wednesday 25 November 2009
non-functionalrequirements
“Equal rights amendment”
Create value vs Protect value
Wednesday 25 November 2009
“Enterprise” Triangle
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
“Enterprise” Triangle
Project
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
“Enterprise” Triangle
Project
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
“Enterprise” Triangle
Project Operations
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
“Enterprise” Triangle
Project Operations
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
“Enterprise” Triangle
Project Operations
Iron Triangle
Wednesday 25 November 2009
Usually only the left triangle is discussed, but it complements the second
Traditional CI limits
• Resistance to change
• « if it ain't broken don't fix it »
• Mostly hear as
• « if it works don’t touch it »
• Some even consider it best practice !
• « don’t put things into prod on a friday »
Wednesday 25 November 2009
Why such a resistance
• development = create value
• operations = protect value
• => operations must limit risks
• change = risk
• so we limit the changes
• QED!
Wednesday 25 November 2009
If it’s hard do it more often
Wednesday 25 November 2009
We need to deploy more often. First times will fail, but eventually we will get better at this.The same for systems rebuilding.
Minimize Risks :integration of production• Add to traditional CI
• deployment of VM, OS,packages, middleware
• Production related tests
• infrastructure
• *ilities...
• performance
Wednesday 25 November 2009
CI for systems
Wednesday 25 November 2009
now managing a system is becoming more like writing software. So we will draw the parallel in the next part of the presentationfor Source , Build, Test
CI for systems (2)
Wednesday 25 November 2009
packaging a system, deploying a system
Remember there is no silver bullet
Wednesday 25 November 2009
Typical System Installation in CI
• Download and Install a distribution
• Download and Add some packages
• Run some configuration scripts
• Start software deployment testing
Wednesday 25 November 2009
Problems?
• It’s a one shot operation
• Systems tend to become application friendly
• Is this the same as production? Patches, Versions, ...
• Usually allow all permissions
Wednesday 25 November 2009
Birth / Adoption
Life
Palliative
Death
Life of System
Wednesday 25 November 2009
Projects tend to think most of the birth of the software.But what about the system? We need to care about the running in production, in phase out and shutdown of a system
Examples of systems integrations in CI software
Hudson - PXE Hudson - Vmware
Not interested in the birth
Wednesday 25 November 2009
Current integration of virtual machines in CI software
Evolution of operations
In the beginning there were the dinosaurs, slow and with large delays
Wednesday 25 November 2009
Often IT is viewed as this
Operations evolution (2)
• Then came the monkeys with automated installations and automated deployment
Wednesday 25 November 2009
Then they automated the installation
Clone system
• there is no such things a golden image
• image sprawl
• Cloning = Adoption of a System
Wednesday 25 November 2009
And then tried to clone things
Test in a clone of the production environment
• Large effort
• similar to maintaining a ‘fork’
Wednesday 25 November 2009
Wednesday 25 November 2009
So let’s build it from the same source!
If we want to integrate let’s define what makes up the ‘source’ of our systems
http://jserv.sayya.org/misc/use_source_luke.png
Virtual Machines API• Systems virtualization :
Vserver, Zones, OpenVZ/Virtuozzo, chroots
• Hardware Emulation: VMWare, Xen, Qemu, VirtualBox, KVM
Wednesday 25 November 2009
First part of the source
Language Bindings
• java libvirt / virsh
• ruby-libvirt
• virtualbox SOAP Interface
• vixie
• vmware API
Wednesday 25 November 2009
Example Virtualbox SOAP
require 'soap/wsdlDriver'require 'pp'
WSDL_URL="vboxwebService.wsdl"
soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driversoap.wiredump_dev=STDERR#soap = SOAP::WSDLDriverFactory.new(WSDL_URL).create_rpc_driver("vboxService", "vboxServicePort")#pp soap.methodsvbox=soap.IWebsessionManager_logon({:username => '', :password => ''})puts "Sessions"+vbox.returnvalversion=soap.IVirtualBox_getVersion({:_this => vbox.returnval})puts version.returnvaldisks=soap.IVirtualBox_getHardDisks({:_this => vbox.returnval})diskids=disks.returnvaldiskids.each do |diskid| type=soap.IHardDisk_getType({:_this => diskid }) size=soap.IHardDisk_getLogicalSize({:_this => diskid }) location=soap.IMedium_getLocation({:_this => diskid }) puts diskid+"-"+type.returnval+"-"+size.returnval+location.returnvalend
Wednesday 25 November 2009
Example libvirt
require 'libvirt'
conn = Libvirt::open("qemu:///system") puts conn.capabilities File::open("domain.xml") do |f| conn.createDomainLinux(f.readlines, nil) end
dom = conn.lookupDomainByName("mydomain") dom.suspend dom.resume puts dom.xmlDesc
Wednesday 25 November 2009
libvirt abstracts a lot of these types Xen, Qemu , Virtualbox, Vmware
Control the machine
• Physical machine
• IPMI, Light Out Module, Wake On LAN
• Virtual machine
• use of an API
Wednesday 25 November 2009
Network provisioning
• DNS
• DHCP
• PXE
• TFTP Boot
Wednesday 25 November 2009
Example Cobbler API# Set the distribution of the machine# The distribution was previously imported by importing an installation DVD# Cobbler will detect the possible kernels to boot# And this will also link the TFTP and PXE file necessary to Bootcobbler profile add --name=#{profile} --distro=centos53-i386
# Add the new machine with an IP and Mac address# By the IP Address it knows in which reverse DNS cobbler system add --name=#{name} --ip=#{ip} --mac=#{mac_address}"# Set the DNS domain of the machine (this determines in which zone file it is create)cobbler system edit --name=#{name} --profile=#{profile} --dns-name=puppet1.#{domainname}"
# For linux machines, the kernels are provided with an option # ks=kickstart so that it will start the kickstart installationcobbler system edit --name=#{name} --kickstart=/var/lib/cobbler/kickstarts/#{kickstart}cobbler system edit --name=#{name} --name-servers-search='#{domainname}'
# This finally commits all the changescobbler sync
Wednesday 25 November 2009
Unattended OS Installs
AutoYastPreseed JumpstartJET Kickstart
.....
Wednesday 25 November 2009
Example Kickstart# install_type MUST be firstinstall_type initial_install# start with the minimal required number of packagescluster SUNWCXallcluster SUNWCapache deletecluster SUNWCpcmc deletecluster SUNWCpcmcx deletecluster SUNWCthai deletecluster SUNWClp deletecluster SUNWCnis deletecluster SUNWCppp delete# format the entire disk for Solarisfdisk all solaris all# define how the disk is partitionedpartitioning explicitfilesys rootdisk.s0 6144 /filesys rootdisk.s1 1024 swapfilesys rootdisk.s7 free /state/partition1# install systems as standalonesystem_type standalone# specify patches to installpatch 119281-06 nfs 172.16.64.194:/export/patches# specify packages to installpackage SPROcc add nfs 172.16.64.194:/export/packages
Wednesday 25 November 2009
It’s there under your nose
• After your manual install (system DEV)
• There is often a left over of silent install scripts
• anaconda
• silent install of oracle
• answer files
Wednesday 25 November 2009
OS minimal
Just Enough Operating System
Wednesday 25 November 2009
Instead of a full installation with the kickstart template. Use a minimal install/base install and then continue with config mgt.
configuration management
• chef
• puppet
• cfengine 3
• bcfg2
• lcfg
• smartfrog HP
Wednesday 25 November 2009
Handles also live of the system.
Sample Recipeclass httpd { package { httpd: ensure => latest }
configfile { "/etc/httpd/conf/httpd.conf": source => "/httpd/httpd.conf", mode => 644, require => package["httpd"] }
group { apache: gid => 48 }
user { apache: comment => "Apache", uid => 48, gid => 48, home => "/var/www", shell => "/sbin/nologin" }
service { httpd: running => true, subscribe => [ file["/etc/httpd/conf/httpd.conf"], package["httpd"] ] }}
Wednesday 25 November 2009
Shareable recipesFormal description of what needs to happen instead of custom scripts
• auto documentation
• confidence
• idempotent!
What a config mgt system can do for you
sed ‘s/the/then/g’
Wednesday 25 November 2009
Managing Config Files API
• augeas - RedHat/Fedora
set /files/etc/hosts/01/ipaddr 192.168.0.1set /files/etc/hosts/01/canonical pigiron.example.comset /files/etc/hosts/01/alias[1] pigironset /files/etc/hosts/01/alias[2] piggysave
Wednesday 25 November 2009
An extension for managing config files
Scriptable Appserver
• glassfish, Weblogic Scripting
• Oracle silent installs
Wednesday 25 November 2009
Further automation
Example AppServer API# create an cluster asadmin create-cluster --user admin --host hostname -port 4848 cluster1
# create instance 1 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties "JMX_SYSTEM_CONNECTOR_PORT=8687:IIOP_LISTENER_PORT=3330:IIOP_SSL_LISTENER_PORT=4440:IIOP_SSL_MUTUALAUTH_PORT=5550:HTTP_LISTENER_PORT=1110:HTTP_SSL_LISTENER_PORT=2220" instance1
# create instance 2 asadmin create-instance --user admin --host hostname -port 4848 --cluster cluster1 --nodeagent nodeagent1 --systemproperties “JMX_SYSTEM_CONNECTOR_PORT=8688:IIOP_LISTENER_PORT=3331:IIOP_SSL_LISTENER_PORT=4441:IIOP_SSL_ MUTUALAUTH_PORT=5551:HTTP_LISTENER_PORT=1111:HTTP_SSL_LISTENER_PORT=2221" instance2
# start the cluster asadmin start-cluster --user admin --host hostname --port 4848 cluster1
Wednesday 25 November 2009
Executing scripts• dsh, func, clusterssh, pssh, fabric
• or system calls in ruby/java/..
• exit code = important!
Wednesday 25 November 2009
For the final tweeks, we run shell scripts to make
Make your build self-testing
Wednesday 25 November 2009
Test Driven Automation
• Checks vs Tests
• Monitoring vs Testing
• Administration vs Automation
Wednesday 25 November 2009
checks = automation of things to verify - similar to monitoringtests goes beyond check (human interpretation)
Red green refactor (change of config, change of patch, ....)
TDD Virtual Machines
Wednesday 25 November 2009
Testing if CPU, Memory , Disk are there by booting up a rescueCD and executing a script.
Example DNS TestingFeature: Mass DNS Resolution In order to present a consistent brand image As a System Administrator I want to ensure no domains point away from our main IP
Scenario Outline: Resolve a name to a number Given a hostname of <hostname> Then I should see the IP address 266.266.266.266
Examples: | hostname | | example.org | | www.example.org |
Wednesday 25 November 2009
http://blog.unixdaemon.net/cgi-bin/blosxom.pl/2009/04/06#ruby-dns-testing
Apache Recipe testing
Wednesday 25 November 2009
Thanks to the build doctor! http://www.build-doctor.com/
Testing DeploymentScripts Capistrano
• SMF monitoring scripts
Wednesday 25 November 2009
http://pivotallabs.com/users/jdean/blog/articles/763-testing-capistrano-recipes-with-cucumber
Testing / Monitoring
Cucumber-Nagioshttp://www.slideshare.net/auxesis/monitoring-web-application-behaviour-with-cucumbernagios
Wednesday 25 November 2009
http://holmwood.id.au/~lindsay/2009/02/23/web-app-integration-testing-for-sysadmins-with-cucumber-nagios/
Lots of non
functionaltests
Wednesday 25 November 2009
http://www.acutest.co.uk/acutest/load-testing-services
Maintain a codesystem repository
• Virtual Machine definitions + automation scripts
• Network provisioning (IP, DNS, DHCP) + scripts
• Install Media (DVD, RPM’s) +
• Kickstart Definitions + postinstall scripts
• Config mgt recipes
• Deployment scripts
Wednesday 25 November 2009
Snapshots
vs.
Wednesday 25 November 2009
Git doesn’t handle well large files / rsync worksRsync snapshots (used by timemachine)
File system repositories
Similarities
• html/js/java - scripts & recipes
• RPMS - jar files
• VM’s - war files
• local repositories
Wednesday 25 November 2009
Every commit should be built
Wednesday 25 November 2009
Automate the Build(of the System)
Wednesday 25 November 2009
Good practice. Not only 1 part of it , but all pieces together.
Builders
• maven, ant,
• buildr
• rake, raven
• make
• schmant
• lancet
• sbt, kundo, gant
Wednesday 25 November 2009
There are a lot of ways to combine all this code using one of the builder tools.They know how to handle dependencies well
Funny Tests
“Truck Factor”
“10th floor test”
Wednesday 25 November 2009
The 10th floor test , is a term first used by Steve Traugott from www.infrastructures.org It refers to the ability to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minutes.
Truck Factor - ProjectTruck Factor (definition): "The number of people on your team who have to be hit with a truck before the project is in serious trouble"
triggers
• changes in VM definition
• changes in IP, DNS, ..
• changes in OS, packages
• changes in recipes
• changes in scripts
Wednesday 25 November 2009
build ENV = PROD• Development
• Test
• Quality Assurance
• Pre-production
• Production
• Differences in hardware
• Differences in software
• Differences in configuration
Wednesday 25 November 2009
Templates/ Config should be configurable with the environment
Everyone commits every day
Wednesday 25 November 2009
Keep the build fast
Wednesday 25 November 2009
Filesystem Snapshots
Virtual Machine Snapshots
LVM, ZFS, ...
Wednesday 25 November 2009
To speed up builds , intermedia results are stored using snapshots (system + disks)
Continuous vs Nightly
• During the day, recipes only
• Nightly the whole system rebuild
Wednesday 25 November 2009
Package
• koan : creates virtual images
• rpath / rbuilder
• ec2
• SUN forklift
• jumpbox
Wednesday 25 November 2009
Deployment tools
• carpet
• ec2 tools AMI
• Vmware Appliance
Wednesday 25 November 2009
http://www.agileweboperations.com/an-in-depth-look-at-a-carpet-appliance-the-apache-load-balancer/
Carpet= Mixture of puppet with capistrano
Everyone can see the results of the latest build
Wednesday 25 November 2009
Make it easy to get the latest deliverables
deploy on the latest installations
Wednesday 25 November 2009
do you checkin wars for archiving
Hot topics
• Cloud Computing
• Production virtualization
• Continuous deployment
• Required fully automated provisioning
Wednesday 25 November 2009
http://dev2ops.blogspot.com/2009/05/clouds-virtualization-and-continuous.html
Concept of Pipelines
Wednesday 25 November 2009
Cloud perspective
Wednesday 25 November 2009
There is actually very much of similarity with the cloud model
Key integration points
• What environments to combine
• Stable dev vs. Stable prod?
• Stable dev vs. Test prod?
• Test dev vs. Test prod?
• => explosion combinations!
• select the most sensible combination
• be sure to be able to pass easily to the next environment
Wednesday 25 November 2009
Problems/Pitfalls• Infrastructure and software need to be
adapted for easy put into production
• update with downtime (load-balancer, redundant servers, clusters,
• data migrations
Wednesday 25 November 2009
State Transition Process
http://digiorgio.com/blog/?p=239
Wednesday 25 November 2009
If N+1 is in development then N is in integration , and N-1 in staging
Parallel Streams
http://www.agilejournal.com/content/view/865/195/
Wednesday 25 November 2009
Example on how to integrate between different teams.
It’s all about flow here!
Development Sysadmins
Wednesday 25 November 2009
Development Sysadmins
DEVOPS
Wednesday 25 November 2009
More?
Email: [email protected] Twitter: @patrickdebois
http://[email protected]
IEEE-paper: http://www.jedi.be/presentations/IEEE-Agile-Infrastructure.pdf
http://www.jedi.be/blog
Wednesday 25 November 2009
Wednesday 25 November 2009
So the next time you’re building a fun project
Wednesday 25 November 2009
Remember the sysadmins and guys from operations
We reboot with pride
Wednesday 25 November 2009
And remember to reboot with pride
Questions?
Wednesday 25 November 2009
Wednesday 25 November 2009
Thanks for listening.