configuration as code in jenkins. what's new? nov 2016

Configuration as Code in Jenkins What's new?Oleg Nenashev (@oleg_nenashev),Jenkins Developer

CloudBees, Inc.

DevOps StockholmNov 28, 2016

2© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

PhD, Hardware EngineeringR&D/Automation in HW and Embedded

Core Team Member, Jenkins projectJenkins Developer, CloudBees

• CloudBees Jenkins Platform• Support and consulting

Contributor, LibreCores project

About me

@oleg_nenashevoleg-nenashevonenashev

3© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps 3

Oleg’s “Hall of Shame”(c)

Also:• Jenkins Core• Remoting –

Communication Layer for Agents

• Windows Service Wrapper


Do know about Jenkins?

About you


1. The most popular CI automation server in the world

2. It’s a framework• Swiss Army Knife of Automation

3. It’s open source• Can be customized

4. Commercial support is available

Who is Mr. Jenkins?


Jenkins is not only about Continuous Integration

Continuous Integration

Continuous Delivery

DevOps, System Administration

Accounting automation

Home Automation

CRON + WebUI


Do you use Jenkins?

About you


Jenkins is Popular

http://stats.jenkins-ci.org/jenkins-stats/

• Common: ~70% of teams use Jenkins

• Almost 150k installations• ~300 active committers

(over last 6 months)• ~3 new plugins every week• ~50 plugin releases every

week

Hudson Jenkins


NEW WEBSITE

NOW IT’S A CD AUTOMATION SERVER

FOCUS ON USER EXPERIENCE

UI IMPROVEMENTS

SECURITY IS ENABLED BY DEFAULT

PIPELINE AS CODE

Jenkins 2

http://bit.ly/jenkins20Stats:• Released on April 2016• LTS line – July 2016• 50% adoption over six months


CI – de-facto standardNew buzzwords

• Continuous Delivery• DevOps• Microservices

Cloud-native systemsCentralization/StandardizationAutomation becomesmission-critical

Automation System trends

No test engineers

What was wrong?


Classic Approaches• Shell script steps to execute custom logic• Matrix Project – multi-configuration jobs• Delivery Pipeline, MultiJob – defining build flows• Parameterized Trigger / Copy Artifact - parallelize

tests and to collect results• Custom Tools plugin to manage tools• Throttle Concurrent Builds and Lockable

Resources to dispatch the access to shared resources• Naginator plugin to restart the builds on-demand• …


What if we combine them?

13


Classic Approaches

Spaghetti Automation

• Many job dependencies

• Difficult build flow tracking

• Complex configurations

• Duplication

• Expensive maintenance


ü Storage together with the projectü Tracking together with the projectü Tests together with the project

…. as code?Configuration…Documentation…Pipeline… Everything…


Configuration as Code in Jenkins

Job ConfigurationJenkins system configuration


Jenkins Configuration… as Code

External Tools

Jenkins CLI and REST

API

python-jenkins

jenkins-client (java)

Configuration Management

systems

Ansible, Chef

Docker

Solutions within Jenkins

project

Groovy Boot Hooks

Scriptler Plugin

SCM Sync Configuration

UnsafeJust examples…


Jenkins 2 & External Configuration Management


Security by default in Jenkins 2Known issues…• chef-cookbooks/jenkins (Issue #466)• puppet-jenkins (Issue #575 )• python-jenkins• …

Fixed• ansible-jenkins (not all implementations)• jenkinsci/docker (for a standard image)

Jenkins 2 & External Configuration Management


So, what’s new?


Something in progressPipeline-alike Configuration as Code plugin• Groovy DSL• Source: ▸ https://github.com/jenkinsci/system-config-dsl-plugin

• Started by Kohsuke Kawaguchi in 2015• Status: pre-alpha


What about System-level?Jenkins system from a single definition


What’s new?No **standard** solution…

But t


What about Jenkins Jobs?


Configuration as Code for jobs

Job DSL

Job Builder Plugin

Jenkins PipelinePipeline is offered by default in Jenkins 2.0

…

Groovy

YAML

Groovy


Pipeline is a…

ØGroovy DSL for automatic flow description *• Alternatives – Job DSL, Jenkins Job Builder

Ø Job type in Jenkins (Pipeline Job)• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations

ØEcosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2


Example

// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"

} http://bit.ly/pipeline-tutorial


Pipeline. Useful FeaturesRobustness against restarts of Jenkins master

Robustness against network disconnects• sh() steps are based on the Durable Task plugin• Jenkins continues execution once the node reconnects

Parallelization across multiple nodes:• It can be achieved via combination of parallel() and node() steps• No need to use the trigger/copy chain (but you can do it)

Ability to store the shared logic in standalone Pipeline libraries

Visualization: Pipeline Stage View, Blue Ocean


Stage View


Blue Ocean


Example 2. Running on two nodes in parallel…stage ‘test'parallel ‘board1' : {node(‘linux && board1') {unstash 'source'sh "./bin/runTest --target=board1"step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml'])

}}, ’board2': {node(‘linux && board2') ) {unstash 'source'sh "./bin/runTest --target=board2"step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml'])

}}


Lockable Resources plugin is integrated with PipelineThrottle Concurrent Builds plugin is not integrated, license limitations may be a problem

Resource management

echo 'Starting'node('board1') {stage(’Compile')// Build project// ...stage(’Run tests')lock(env.NODE_NAME + "-board1") {// Run something on the board// ...

}}


FailoverPipeline allows re-running unreliable stages on other nodes.

System failure

Node #1 Node #2. . .

. . .

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 255")) {// Fatal errorfail("Test run failed")

}}

}

Try all possible boards:


Docker PluginDocker Pipeline PluginDocker Custom Build Environment Plugin…Yet Another Docker Plugin

Example: Docker plugins


Example. Docker-based tools and PipelineDocker Pipeline Plugin:

• You can share workspaces• External Workspace Manager

• You can create build chains

See also: Kubernetes Pipeline Plugin


Example. CD for Jenkins IRC Botdef imageName = 'jenkinsciinfra/ircbot’

node('docker') {

checkout scm

sh 'git rev-parse HEAD > GIT_COMMIT'

commit = readFile('GIT_COMMIT').take(6)

def imageTag = "build${commit}"

stage('Build ircbot’) {

withMavenEnv

(["BUILD_NUMBER=${env.BUILD_NUMBER}:${commit}"]) {

sh 'make bot' // Make invokes Maven

}}

stage ('Build container’) {

def whale = docker.build("${imageName}:${imageTag}”)

}

stage 'Deploy container') {

whale.push()

} }https://github.com/jenkins-infra/ircbot

Docker Registry

Server

Jenkins

SCM

Puppet


Pipeline features. Interactive step// Buildtheproject,blastittotheboards//Preparetheruninthetestmanagementsystem

//Sendthenotificationtotesters.Thenwaitfortheinputecho’Automatictestspassed’hipchatSend('@QATeam Thebuild' + buildVersion + 'hasbeenstaged:' + stagingBoard

+ '.PleaseTestitandreporttheresultbackto' + build.url+ '.TestLink runishere:' + testLinkRunURL)

inputmessage:'Havemanualspot-checkspassed?Readytorelease?'

//Proceedonthesuccessfulpath

mailto:"[email protected]",subject:"DreamProduct- buildisreleased",body:"Hithere,Thenewbuildoftheproducthasbeenreleased.Link:" +artifactoryLink

~~~~~~Code~~~~~~

~~~~~~Code~~~~~~

Too Complex?• More syntax sugar• Pipeline Model Definition• Snippet Generator


Syntax Sugar - @Symbol

$class is not required anymore… almost•Plugins need to be adapted

It simplifies syntax of Pipeline and JobDSL


Syntax Sugar - @SymbolWas:

Now:


Syntax sugar on the top of Pipeline• “pipeline { }” closure

Provides common configuration sections in Pipeline• Some static analysis of the configuration• Less flexibility• Simplified case like error handling

More info:• https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki

42

Pipeline Model Definition


Example. Pipeline Model Definitionpipeline { agent label:"generic-linux"tools { maven "Maven 3.3.9"jdk "Oracle JDK 8u40"

} stages { stage("build") { steps { sh 'mvn clean install -Dmaven.test.failure.ignore=true' }

} }post { always { junit "path/to/xml" } failure { mail to:"[email protected]", subject:"FAILURE:${currentBuild.fullDisplayName}",

body: "${currentBuild.url}" } }

}

Jenkins Pipeline Library Management


load() – loading of Groovy code from a fileCPS Global Library•Comes as a part of Pipeline plugin set•Creates Git repo within Jenkins•Defines global variables and classes

Pipeline Remote Loader•https://github.com/jenkinsci/workflow-remote-loader-plugin•Loads Pipeline from Git/GitHub•The plugin logic is implemented as Pipeline

Was

45


Now. Pipeline Shared Libraries /

Library Management in Pipeline


Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸Jenkins system-wide▸Folder▸Job


Pipeline Library Definition [1/2]


Pipeline Library Definition [2/2]


Using Pipeline Libraries


More examples:https://github.com/jenkinsci/pipeline-

examples


Pipeline. What’s next?

• Stability / Fault tolerance• UI/UX, Blue Ocean• Syntax improvements• (?) Freestyle => Pipeline Converter


Pipeline. Dev toolsAvailable: •Minimal IDE: Auto-completion•Minimal static analysis•Documentation browser for steps•Library manager

Missing:•Debugger•Real-time code deployment from IDE•Real-time approval of Script Security•Freestyle => Pipeline Converter


• Pipeline greatly simplifies task parallelization and failover within a single build

• Pipeline saves much time on automation system maintenance• Pipeline is a programming languages, high barrier to entry• Pipeline still lacks Dev Tools and integrations, but it evolves

Pipeline. Lessons learnt


Summary

1. Jenkins is an effective automation framework

2. Jenkins offers Configuration-as-Code for System configs and Jobs

3. C-as-C for Jenkins System configurations is far from ideal

4. C-as-C for Jobs is not ideal as well, but it evolves quickly

• Especially Pipeline ecosystem


1. Use configuration as code where it is possible2. If you use Jenkins…

1. Consider upgrading to Jenkins 22. Try Jenkins Pipeline3. Organize your system management scripts

3. If you use Pipeline…1. Try Pipeline model definition2. Try new Pipeline libraries

Takeaways


Jenkins project:• Website: https://jenkins.io

Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples• Model definition: https://github.com/jenkinsci/pipeline-model-definition-plugin

CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

Useful Links

Software at the speed of ideas

THANK YOU!www.cloudbees.com

Questions?Offline: [email protected]

oleg-nenashev

oleg_nenashev