configuration as code in jenkins. what's new? nov 2016

Configuration as Code in Jenkins What's new?Oleg Nenashev (@oleg_nenashev),Jenkins Developer

CloudBees, Inc.

DevOps StockholmNov 28, 2016

2© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

PhD, Hardware EngineeringR&D/Automation in HW and Embedded

Core Team Member, Jenkins projectJenkins Developer, CloudBees

• CloudBees Jenkins Platform• Support and consulting

Contributor, LibreCores project

About me

@oleg_nenashevoleg-nenashevonenashev

3© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps 3

Oleg’s “Hall of Shame”(c)

Also:• Jenkins Core• Remoting –

Communication Layer for Agents

• Windows Service Wrapper

4© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Do know about Jenkins?

About you

5© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

1. The most popular CI automation server in the world

2. It’s a framework• Swiss Army Knife of Automation

3. It’s open source• Can be customized

4. Commercial support is available

Who is Mr. Jenkins?

6© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Jenkins is not only about Continuous Integration

Continuous Integration

Continuous Delivery

DevOps, System Administration

Accounting automation

Home Automation

CRON + WebUI

7© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Do you use Jenkins?

About you

8© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Jenkins is Popular

http://stats.jenkins-ci.org/jenkins-stats/

• Common: ~70% of teams use Jenkins

• Almost 150k installations• ~300 active committers

(over last 6 months)• ~3 new plugins every week• ~50 plugin releases every

week

Hudson Jenkins

9© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

NEW WEBSITE

NOW IT’S A CD AUTOMATION SERVER

FOCUS ON USER EXPERIENCE

UI IMPROVEMENTS

SECURITY IS ENABLED BY DEFAULT

PIPELINE AS CODE

Jenkins 2

http://bit.ly/jenkins20Stats:• Released on April 2016• LTS line – July 2016• 50% adoption over six months

10© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

CI – de-facto standardNew buzzwords

• Continuous Delivery• DevOps• Microservices

Cloud-native systemsCentralization/StandardizationAutomation becomesmission-critical

Automation System trends

No test engineers

What was wrong?

12© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Classic Approaches• Shell script steps to execute custom logic• Matrix Project – multi-configuration jobs• Delivery Pipeline, MultiJob – defining build flows• Parameterized Trigger / Copy Artifact - parallelize

tests and to collect results• Custom Tools plugin to manage tools• Throttle Concurrent Builds and Lockable

Resources to dispatch the access to shared resources• Naginator plugin to restart the builds on-demand• …

13© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

What if we combine them?

13

14© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Classic Approaches

Spaghetti Automation

• Many job dependencies

• Difficult build flow tracking

• Complex configurations

• Duplication

• Expensive maintenance

15© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

ü Storage together with the projectü Tracking together with the projectü Tests together with the project

…. as code?Configuration…Documentation…Pipeline… Everything…

16© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Configuration as Code in Jenkins

Job ConfigurationJenkins system configuration

17© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Jenkins Configuration… as Code

External Tools

Jenkins CLI and REST

API

python-jenkins

jenkins-client (java)

Configuration Management

systems

Ansible, Chef

Docker

Solutions within Jenkins

project

Groovy Boot Hooks

Scriptler Plugin

SCM Sync Configuration

UnsafeJust examples…

18© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Jenkins 2 & External Configuration Management

19© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Security by default in Jenkins 2Known issues…• chef-cookbooks/jenkins (Issue #466)• puppet-jenkins (Issue #575 )• python-jenkins• …

Fixed• ansible-jenkins (not all implementations)• jenkinsci/docker (for a standard image)

Jenkins 2 & External Configuration Management

20© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

So, what’s new?

21© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Something in progressPipeline-alike Configuration as Code plugin• Groovy DSL• Source: ▸ https://github.com/jenkinsci/system-config-dsl-plugin

• Started by Kohsuke Kawaguchi in 2015• Status: pre-alpha

22© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

What about System-level?Jenkins system from a single definition

23© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

What’s new?No **standard** solution…

But t

24© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

What about Jenkins Jobs?

25© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Configuration as Code for jobs

Job DSL

Job Builder Plugin

Jenkins PipelinePipeline is offered by default in Jenkins 2.0

…

Groovy

YAML

Groovy

26© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline is a…

ØGroovy DSL for automatic flow description *• Alternatives – Job DSL, Jenkins Job Builder

Ø Job type in Jenkins (Pipeline Job)• Job definitions can be stored in SCM• Automatic job creation from repositories and organizations

ØEcosystem for automation as code• ~20 dedicated plugins• Integration in many other plugins• Suggested by default in Jenkins 2

27© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Example

// Run on a node having the “my_fpga” labelnode("linux && ml509") { git url:"http://github.com/myorg/myproject.git"sh "make all"

} http://bit.ly/pipeline-tutorial

28© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline. Useful FeaturesRobustness against restarts of Jenkins master

Robustness against network disconnects• sh() steps are based on the Durable Task plugin• Jenkins continues execution once the node reconnects

Parallelization across multiple nodes:• It can be achieved via combination of parallel() and node() steps• No need to use the trigger/copy chain (but you can do it)

Ability to store the shared logic in standalone Pipeline libraries

Visualization: Pipeline Stage View, Blue Ocean

29© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Stage View

30© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Blue Ocean

31© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps 31

32© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Example 2. Running on two nodes in parallel…stage ‘test'parallel ‘board1' : {node(‘linux && board1') {unstash 'source'sh "./bin/runTest --target=board1"step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml'])

}}, ’board2': {node(‘linux && board2') ) {unstash 'source'sh "./bin/runTest --target=board2"step([$class:'JUnitResultArchiver', testResults:'**/test-results/*.xml'])

}}

33© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Lockable Resources plugin is integrated with PipelineThrottle Concurrent Builds plugin is not integrated, license limitations may be a problem

Resource management

echo 'Starting'node('board1') {stage(’Compile')// Build project// ...stage(’Run tests')lock(env.NODE_NAME + "-board1") {// Run something on the board// ...

}}

34© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

FailoverPipeline allows re-running unreliable stages on other nodes.

System failure

Node #1 Node #2. . .

. . .

for (def board : boards) {echo "trying board " + board;try {node(board) {checkout scmsh ‘./bin/run.sh’ // Call passed => DONEbreak;

}} catch (Exception ex) {if (ex.message.contains

("exit code 255")) {// Fatal errorfail("Test run failed")

}}

}

Try all possible boards:

35© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Docker PluginDocker Pipeline PluginDocker Custom Build Environment Plugin…Yet Another Docker Plugin

Example: Docker plugins

36© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Example. Docker-based tools and PipelineDocker Pipeline Plugin:

• You can share workspaces• External Workspace Manager

• You can create build chains

See also: Kubernetes Pipeline Plugin

37© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Example. CD for Jenkins IRC Botdef imageName = 'jenkinsciinfra/ircbot’

node('docker') {

checkout scm

sh 'git rev-parse HEAD > GIT_COMMIT'

commit = readFile('GIT_COMMIT').take(6)

def imageTag = "build${commit}"

stage('Build ircbot’) {

withMavenEnv

(["BUILD_NUMBER=${env.BUILD_NUMBER}:${commit}"]) {

sh 'make bot' // Make invokes Maven

}}

stage ('Build container’) {

def whale = docker.build("${imageName}:${imageTag}”)

}

stage 'Deploy container') {

whale.push()

} }https://github.com/jenkins-infra/ircbot

Docker Registry

Server

Jenkins

SCM

Puppet

38© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline features. Interactive step// Buildtheproject,blastittotheboards//Preparetheruninthetestmanagementsystem

//Sendthenotificationtotesters.Thenwaitfortheinputecho’Automatictestspassed’hipchatSend('@QATeam Thebuild' + buildVersion + 'hasbeenstaged:' + stagingBoard

+ '.PleaseTestitandreporttheresultbackto' + build.url+ '.TestLink runishere:' + testLinkRunURL)

inputmessage:'Havemanualspot-checkspassed?Readytorelease?'

//Proceedonthesuccessfulpath

mailto:"dream_team@cloudbees.com",subject:"DreamProduct- buildisreleased",body:"Hithere,Thenewbuildoftheproducthasbeenreleased.Link:" +artifactoryLink

~~~~~~Code~~~~~~

~~~~~~Code~~~~~~

Too Complex?• More syntax sugar• Pipeline Model Definition• Snippet Generator

40© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Syntax Sugar - @Symbol

$class is not required anymore… almost•Plugins need to be adapted

It simplifies syntax of Pipeline and JobDSL

41© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Syntax Sugar - @SymbolWas:

Now:

42© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Syntax sugar on the top of Pipeline• “pipeline { }” closure

Provides common configuration sections in Pipeline• Some static analysis of the configuration• Less flexibility• Simplified case like error handling

More info:• https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki

42

Pipeline Model Definition

43© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps 43

Example. Pipeline Model Definitionpipeline { agent label:"generic-linux"tools { maven "Maven 3.3.9"jdk "Oracle JDK 8u40"

} stages { stage("build") { steps { sh 'mvn clean install -Dmaven.test.failure.ignore=true' }

} }post { always { junit "path/to/xml" } failure { mail to:"me@example.com", subject:"FAILURE:${currentBuild.fullDisplayName}",

body: "${currentBuild.url}" } }

}

Jenkins Pipeline Library Management

45© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

load() – loading of Groovy code from a fileCPS Global Library•Comes as a part of Pipeline plugin set•Creates Git repo within Jenkins•Defines global variables and classes

Pipeline Remote Loader•https://github.com/jenkinsci/workflow-remote-loader-plugin•Loads Pipeline from Git/GitHub•The plugin logic is implemented as Pipeline

Was

45

46© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Now. Pipeline Shared Libraries /

Library Management in Pipeline

47© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline Shared Libraries• Libs are located in SCM• Versioning• Definition for scopes:▸Jenkins system-wide▸Folder▸Job

48© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline Library Definition [1/2]

49© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline Library Definition [2/2]

50© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Using Pipeline Libraries

51© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

More examples:https://github.com/jenkinsci/pipeline-

examples

52© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline. What’s next?

• Stability / Fault tolerance• UI/UX, Blue Ocean• Syntax improvements• (?) Freestyle => Pipeline Converter

53© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Pipeline. Dev toolsAvailable: •Minimal IDE: Auto-completion•Minimal static analysis•Documentation browser for steps•Library manager

Missing:•Debugger•Real-time code deployment from IDE•Real-time approval of Script Security•Freestyle => Pipeline Converter

54© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

• Pipeline greatly simplifies task parallelization and failover within a single build

• Pipeline saves much time on automation system maintenance• Pipeline is a programming languages, high barrier to entry• Pipeline still lacks Dev Tools and integrations, but it evolves

Pipeline. Lessons learnt

55© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Summary

1. Jenkins is an effective automation framework

2. Jenkins offers Configuration-as-Code for System configs and Jobs

3. C-as-C for Jenkins System configurations is far from ideal

4. C-as-C for Jobs is not ideal as well, but it evolves quickly

• Especially Pipeline ecosystem

56© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

1. Use configuration as code where it is possible2. If you use Jenkins…

1. Consider upgrading to Jenkins 22. Try Jenkins Pipeline3. Organize your system management scripts

3. If you use Pipeline…1. Try Pipeline model definition2. Try new Pipeline libraries

Takeaways

57© 2016 CloudBees, Inc. All Rights Reserved.@oleg_nenashev, @cloudbees, #SthlmDevOps

Jenkins project:• Website: https://jenkins.io

Pipeline:• Tutorial: https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md• Compatibility: https://github.com/jenkinsci/pipeline-plugin/blob/master/COMPATIBILITY.md• Examples: https://github.com/jenkinsci/pipeline-examples• Model definition: https://github.com/jenkinsci/pipeline-model-definition-plugin

CloudBees website:• Website: https://www.cloudbees.com• CloudBees Jenkins Platform: https://www.cloudbees.com/products/cloudbees-jenkins-platform

Useful Links

Software at the speed of ideas

THANK YOU!www.cloudbees.com

Questions?Offline: onenashev@cloudbees.com

oleg-nenashev

oleg_nenashev