ci and cd across the enterprise with jenkins (devops.com nov 2014)

©2014 CloudBees, Inc. All Rights Reserved

CI and CD Across the Enterprise with Jenkins

Cyrille Le Clerc, CloudBees


About Me

Cyrille Le Clerc

@cyrilleleclerc

Open Source

CTO

DevOps, Infra as Code, Con9nuous Delivery

Product Management


Agenda

•  Con9nuous Delivery & Jenkins Workflow •  Scaling Jenkins •  Con9nuous Delivery with Jenkins and Puppet and Chef


Continuous Delivery


Continuous Delivery

•  “Reliable So+ware Releases through Build, Test, and Deployment Automa=on”

•  DONE = Shippable into Produc9on


Continuous Integration Steps

Con$

nuou

s Integra$o

n Source Code

Quality Analysis

Compile &Unit Tests

Binaries


Continuous Delivery Steps

Con$

nuou

s Delivery

DEV

DEV

OPS

QA

Source Code

Quality Analysis

Compile &Unit Tests

Binaries

Perfs Environment

Func Test Environment

UAT Environment

Production Environment

GO LIVE!

Release!

Staging Environment


Continuous Delivery Pipeline


Compile &Unit Tests

Perfs Tests

Selenium Tests Deploy UAT

Sonar Analysis

Deploy Staging Deploy Production


Jenkins: Hub of Continuous Delivery

•  Connect the dots

•  It is the process Source Code

Perfs Environment

Func Test Environment

UAT Environment

Binaries

Quality Analysis

Compile &Unit Tests


CD Pipeline è Workflow


Compile &Unit Tests

Perfs Tests


Sonar Analysis



What we need

•  Complex pipelines involving mul9ple stages •  Non-‐sequen$al logic such as loops, forks … •  Long-‐running builds must survive outages •  Interac$ons with humans including pauses, input •  Restartable builds in case of transient errors •  Reusable defini$ons to avoid duplica9on •  Comprehensible scripts with one clear defini9on


Workflow until today

•  Job chaining

•  Build Flow Plugin

•  Build Pipeline Plugin

•  Build Delivery Pipeline


Workflow until today

•  Many atomic jobs •  Hard to share variables/state between jobs •  Limited logic •  Mix build triggers, parameterized build …


Build Flow Plugin

•  Did have scriptability and extensibility •  Did not address configura9on “sprawl” •  Disjointed view of what really ran •  No ability to survive restarts •  Almost good enough but could not go further


New Workflow Syntax


Demo time! https://github.com/cyrille-leclerc/spring-petclinic/blob/petclinic-1.0.0-clc/src/main/jenkins/workflow.groovy


CD Pipeline è Workflow


Compile &Unit Tests

Perfs Tests


Sonar Analysis



Key features

•  En9re flow is one concise Groovy script –  For loops, try-‐finally, fork-‐join …

•  Can restart Jenkins while flow is running •  Allocate slave nodes and workspaces

–  As many as you want, when you want

•  Stages throYle concurrency of builds •  Human input/approval integrated into flow •  Standard project concepts: SCM, ar9facts, plugins


Groovy DSL vs. Graphical Workflow

•  Familiar control flow construc9on •  Familiar “tools” for building abstrac9ons

Func9ons, classes, variables, …

•  Workflow in version control •  As opposed to:

Graphical workflow designer


Project setup

•  One workflow is defined as a job •  Single script for all steps •  Build triggers & parameters like regular projects •  Each workflow execu9on is a regular Jenkins build displayed in regular Jenkins views

•  Graphical visualiza9on of actual build possible Not of visualiza9on job defini9on but of build execu9on


Non-sequential logic

try {} catch {}


Interaction with humans


Restartable build / checkpoint


Visualization


Stages

•  Semaphore •  Visualiza9on


Comprehensible scripts


Scaling with Jenkins


•  How do I scale Jenkins from 10 to 1000 jobs? Projects? Users?

•  How do I comply with corporate

rules?

•  How do I keep Jenkins flexible for project teams?

Problem Statement: Scale!


Typical Jenkins installations

Let’s think about this for a few minutes


The top down “structured” and “lets grow vertically” approach

5 projects 0 Slaves

25 projects 5 Slaves




Vertical Scaling issues

When is the next failure? Single point of failure


The “organic” agile growth – “let each group do what they want to” approach

Horizontally… virally


•  Share executors between masters –  Not possible today

Horizontal Issues: I Would Like to…

Master-‐b

Master-‐a

Master-‐c


•  Share security information –  Security realm and roles

•  Push configurations & upgrades

Master-‐b

Master-‐a

Master-‐c

Horizontal Issues: I Would Like to…


Jenkins Opera9ons Center

oc.cloudbees.com Sample Topology

Master-‐b

Master-‐a

Master-‐c

Shared Roles , Security Realm and versions

AYached-‐master client



pushed to client master

pushed to client master


Demo time!


Continuous Delivery traceability with Jenkins and puppet and Chef

xxx Environment


Puppet environments

Prod style environment: –  Master & agents

Dev Environment

Puppet Agentpetclinic.war

Application Serverwar file + puppet manifests

& modules

puppet apply

“Production style” Environment

Puppet Master



& modules

Dev environment: –  Standalone agents + “puppet apply”


Test environment with Puppet and Vagrant

Dev Environment



& modules

vagrant provisionpuppet apply

Vagrant to create “clean” puppet managed servers


CI with Jenkins and Vagrant and Puppet

Jenkins builds war file and puppet modules and deploys via vagrant

puppet deployment

modulejava code Dev Environment



& modules



CI with Jenkins and Vagrant and Puppet

Jenkins builds war file and puppet modules and deploys via vagrant

puppet deployment




& modules


puppet deployment




& modules


notification


Jenkins – Puppet Integration


Demo time! https://github.com/CloudBees-community/vagrant-puppet-petclinic/tree/vagrant-puppet-petclinic-0.1.0 jenkins-workflow-with-vagrant-and-puppet.groovy: https://gist.github.com/cyrille-leclerc/7f621c341e193622c844


Jenkins & Puppet

•  Jenkins Deployment No9fica9on Plugin –  hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin

•  Puppet module for Jenkins tracking –  hYps://github.com/jenkinsci/puppet-‐jenkinstracking

•  Puppet Labs plugin for Jenkins –  hYps://wiki.jenkins-‐ci.org/display/JENKINS/Puppet+Plugin

•  CloudBees DevOps page –  hYps://developer.cloudbees.com/bin/view/Main/DevOps

•  This demo –  hYps://github.com/CloudBees-‐community/vagrant-‐puppet-‐petclinic –  Jenkins Workflow with Vagrant and Puppet: hYps://gist.github.com/cyrille-‐leclerc/7f621c341e193622c844


Jenkins & Chef?

•  Similar concepts •  Jenkins Deployment No9fica9on Plugin

–  hYps://wiki.jenkins-‐ci.org/display/JENKINS/Deployment+No9fica9on+Plugin

•  Puppet module for Jenkins tracking –  hYps://github.com/jenkinsci/puppet-‐jenkinstracking

•  Jenkins Chef Tracking –  hYps://wiki.jenkins-‐ci.org/display/JENKINS/Chef+Tracking+Plugin

•  CloudBees DevOps page –  hYps://developer.cloudbees.com/bin/view/Main/DevOps


Conclusion

•  Jenkins workflow to build Con9nuous Delivery pipelines

•  Jenkins Opera9ons Center by CloudBees to scale Jenkins

•  Puppet & Chef deployments can be integrated in Jenkins

ci and cd across the enterprise with jenkins (devops.com nov 2014)

Technology

rights reserved jenkins

rights reserved workflow

rights reserved ci

production2014 cloudbees

rights reserved demo

continuous delivery

continuous delivery

hub of continuous delivery