building jenkins pipelines at scale
DESCRIPTION
Talk given at Jenkins User Conference Europe 2014 in BerlinTRANSCRIPT
![Page 1: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/1.jpg)
.
.
.
Jenkins User Conference Europe
.#jenkinsconf
.
Building Pipelinesat Scale
Julien Pivottoinuits.eu
25th June 2014
![Page 2: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/2.jpg)
.
.
.
Jenkins User Conference Europe
.#jenkinsconf
.
..Building Pipelines at Scale
IntroductionWho’s whoPipelines
PuppetWhyHowConclusion
PluginsWhyHow
Conclusion
![Page 3: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/3.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Who’s who
.
.. Julien Pivotto
Open-Source Consultant at inuits.eu
Linux Systems AdministratorFOSS Defender for many years� DevOpsPuppet user since 2011Doing also some Ruby & Python
![Page 4: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/4.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Who’s who
.
.. inuits
Open-Source Consultancy company50+ people in 4 countries (.be, .ua, .nl & .cz)Doing both Development & Systems AdministrationA lot of domains: Linux, Web, Databases, Monitoring…Early DevOps practitioners
![Page 5: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/5.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Who’s who
.
.. MediaMosa
Drupal-based Digital Asset Management systemOpen-sourceStore assetsTranscode videosCreate, extract and manage metadata using open standards:Dublin Core, Qualified DC, IEEE/LOM, CZPWebservice oriented
![Page 6: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/6.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Who’s who
.
.. MediaSalsa: MediaMosa As a Service
A turn-key MediaMosa solutionA few environments:
Dev/Uat/Prod environmentsCustomer-specific environments
One backend per environmentMultiple frontends per backend
= A lot of Drupal sites
![Page 7: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/7.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Definition of pipelines
..
A pipeline is a chain of Jenkins jobs that are run to fetch,compile, package, run tests and deploy an application (buildpipeline plugin).
Pipeline vs large (or independent) jobs:Separation between different tasksOverview of tests, etc…Deployment depends on testing
![Page 8: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/8.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Definition of pipelines
..
A pipeline is a chain of Jenkins jobs that are run to fetch,compile, package, run tests and deploy an application (buildpipeline plugin).
Pipeline vs large (or independent) jobs:Separation between different tasksOverview of tests, etc…Deployment depends on testing
![Page 9: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/9.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Example: a puppet pipeline
![Page 10: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/10.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Inside the pipeline
CheckoutStyleSyntaxCompileUnit testsIntegration testsPackagingRegression testsDeployment / Delivery…
![Page 11: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/11.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Ok, let’s use pipelines!
Let’s make a pipeline for our puppet code
…and one for our puppet code in dev environment…and one for our puppet code in uat environmentLet’s make a pipeline for backend app…and one for backend app in dev environment…and one for backend app in uat environmentLet’s make 5 pipelines for frontend apps…and 5 for frontends apps in dev environment…and 5 for frontends apps in uat environment
![Page 12: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/12.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Ok, let’s use pipelines!
Let’s make a pipeline for our puppet code…and one for our puppet code in dev environment…and one for our puppet code in uat environment
Let’s make a pipeline for backend app…and one for backend app in dev environment…and one for backend app in uat environmentLet’s make 5 pipelines for frontend apps…and 5 for frontends apps in dev environment…and 5 for frontends apps in uat environment
![Page 13: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/13.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Ok, let’s use pipelines!
Let’s make a pipeline for our puppet code…and one for our puppet code in dev environment…and one for our puppet code in uat environmentLet’s make a pipeline for backend app
…and one for backend app in dev environment…and one for backend app in uat environmentLet’s make 5 pipelines for frontend apps…and 5 for frontends apps in dev environment…and 5 for frontends apps in uat environment
![Page 14: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/14.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Ok, let’s use pipelines!
Let’s make a pipeline for our puppet code…and one for our puppet code in dev environment…and one for our puppet code in uat environmentLet’s make a pipeline for backend app…and one for backend app in dev environment…and one for backend app in uat environment
Let’s make 5 pipelines for frontend apps…and 5 for frontends apps in dev environment…and 5 for frontends apps in uat environment
![Page 15: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/15.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Ok, let’s use pipelines!
Let’s make a pipeline for our puppet code…and one for our puppet code in dev environment…and one for our puppet code in uat environmentLet’s make a pipeline for backend app…and one for backend app in dev environment…and one for backend app in uat environmentLet’s make 5 pipelines for frontend apps…and 5 for frontends apps in dev environment…and 5 for frontends apps in uat environment
![Page 16: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/16.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
.. How to scale?
First attempt: clone jobs in the uiDozens of jobs to clone & editHuman intervention = mistakes
Second attempt: clone jobs xml filesgrep & cp & sed…Crappy bash scripting is crappy
Mmmh…we need something else…
![Page 17: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/17.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 18: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/18.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click
click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 19: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/19.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click click
click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 20: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/20.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click click click click click
click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 21: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/21.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click
click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 22: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/22.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Introduction/Pipelines
.
..Let’s change the pipelines now!
click click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click click click click click click click click click click clickclick click click
![Page 23: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/23.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet
.
Puppet
![Page 24: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/24.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Why
.
.. Infrastructure as Code
Puppet/Chef/Cfengine/…same fight:Modelize your infrastructureReproducable platformsFast and reliableDisaster recovery for ”free”
![Page 25: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/25.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Why
.
..What to puppetize
Usually you puppetize:OS installation & configurationApps installation & configurationDB installation & configuration
But you do not automate:Application data
Are Jenkins jobs to be puppetized?…we tried…
![Page 26: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/26.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Why
.
..What to puppetize
Usually you puppetize:OS installation & configurationApps installation & configurationDB installation & configuration
But you do not automate:Application data
Are Jenkins jobs to be puppetized?
…we tried…
![Page 27: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/27.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Why
.
..What to puppetize
Usually you puppetize:OS installation & configurationApps installation & configurationDB installation & configuration
But you do not automate:Application data
Are Jenkins jobs to be puppetized?…we tried…
![Page 28: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/28.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
.. Modules structure
Using the Jenkins upstream moduleAnd creating a specific jenkinsjobs module
Difficult to shareMainly templates
![Page 29: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/29.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
.. Structure of the module
manifestsjob.pp (definition; file)packages.pp (packages needed by tests)dashboard.pp (augeas xml to modify jenkins config)pipeline/frontend.pp (definition; jobs)pipeline/backend.pp (definition; jobs)service.pp (exec to reload configuration)
![Page 30: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/30.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..job.pp
Takes a template as parameterCreates xml file and reload jenkins configurationIt has a lot more arguments (project name, next job, etc…)Adds the job to the view (augeas)
![Page 31: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/31.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..service.pp
Reload the Jenkins service
..
java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar-s http://127.0.0.1:8080/ reload-configuration--username "${username}" --password "${password}"
It makes Jenkins unavailable
![Page 32: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/32.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..service.pp
Reload the Jenkins service
..
java -jar /var/cache/jenkins/war/WEB-INF/jenkins-cli.jar-s http://127.0.0.1:8080/ reload-configuration--username "${username}" --password "${password}"
It makes Jenkins unavailable
![Page 33: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/33.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Parameters
.
define jenkinsjobs::pipeline::frontend ($git_repository,$ensure = 'present',$customer = $title,$debian_slave = undef,$dashboard_view = undef,$vhost_docroot = undef,$targets = {},
)
![Page 34: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/34.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Defaults for jobs
.
Jenkinsjobs::Job{ensure => $ensure,git_repository => $git_repository,customer => $customer,assigned_node => $debian_slave,dashboard_view => $dashboard_view,package_name => "frontend-${customer}",vhost_docroot => $_vhost_docroot,custom_modules => 'sites/all/modules/custom',job_type => 'drupal',
}
![Page 35: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/35.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Defaults for promotions
.
Jenkinsjobs::Promotion{ensure => $ensure,customer => $customer,assigned_node => $debian_slave,packaging_job => "frontend-build-${customer}",package_name => "frontend-${customer}",vhost_docroot => $_vhost_docroot,job_name => "frontend-promo-${customer}",
}
![Page 36: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/36.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Jobs definitions
.
jenkinsjobs::job{"frontend-checkout-${customer}":
job_template => 'jj/jobs/checkout.erb',next_job => "frontend-syntax-${customer}",
start_pipeline => true,}
jenkinsjobs::job{"frontend-syntax-${customer}":
job_template => 'jj/jobs/syntax.erb',next_job => "frontend-style-${customer}",
}
![Page 37: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/37.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Job with promotions
.
jenkinsjobs::job{"frontend-promo-${customer}":
job_template => 'jj/jobs/promote.erb',promotions => [
"deploy-${customer}-front-to-uat","deploy-${customer}-front-to-prod","deploy-${customer}-front-to-com-prod",
],}
![Page 38: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/38.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..pipeline.pp
..
Promotions
.
if 'uat' in $targets {jenkinsjobs::promotion {
"deploy-${customer}-front-to-uat":template => 'jj/promotion/uat.erb',debian_repo => 'mediamosa-uat',targets => $targets['viaa-uat'],
}}
![Page 39: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/39.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/How
.
..dashboard.pp
..
Augeas example
.
augeas {"dashboard view $title":
lens => 'Xml.lns',incl => '/var/lib/jenkins/config.xml',context => '/files/var/lib/jenkins/config.xml',changes => [
"set ${augeas_prefix}[last()+1]/owner/#attribute/class \"hudson\"","set ${augeas_prefix}[last()]/owner/#attribute/reference \"../../..\"","set ${augeas_prefix}[last()]/includeStdJobList/#text \"true\"","set ${augeas_prefix}[last()]/name/#text \"${title}\"",
],onlyif =>
"match ${file_prefix}/hudson/views/*/name/#text[.=\"${title}\"] size == 0",notify => Exec['reload-jenkins'],
}
![Page 40: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/40.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelines
Deploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)Track history of pipeline changesHaving separate jobs for each projectReuse puppet template functionalityDeploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in syncAdd ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 41: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/41.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelinesDeploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)
Track history of pipeline changesHaving separate jobs for each projectReuse puppet template functionalityDeploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in syncAdd ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 42: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/42.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelinesDeploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)Track history of pipeline changes
Having separate jobs for each projectReuse puppet template functionalityDeploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in syncAdd ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 43: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/43.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelinesDeploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)Track history of pipeline changesHaving separate jobs for each projectReuse puppet template functionality
Deploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in syncAdd ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 44: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/44.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelinesDeploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)Track history of pipeline changesHaving separate jobs for each projectReuse puppet template functionalityDeploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in sync
Add ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 45: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/45.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Advantages of setting pipelines in Puppet
Easy to deploy new pipelinesDeploy pipelines in the same time as applications (exportedresources)Define pipelines at the same place as applications (puppet)Track history of pipeline changesHaving separate jobs for each projectReuse puppet template functionalityDeploy new Jenkins server in minutes with all jobsKeep several Jenkins (test, prod) in syncAdd ssh keys, git config, etc…in Puppet tooCreate test databases & vhosts in Puppet (exported resources)
![Page 46: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/46.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Puppet/Conclusion
.
..Disadvantages
Play with Augeas & xml (wrong feeling)Makes Jenkins unavailable on changesEvery small change ends up in Puppet
![Page 47: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/47.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins
.
Plugins
![Page 48: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/48.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/Why
.
.. A lighter approach: Jenkins plugins
Remove the Puppet complexityStill having Pipelines in syncMinimize human interventionSpeed-up the pipelinesDo not restart Jenkins each time
![Page 49: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/49.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Plugin #1: Build flow
Creates PipelinesUses Groovy scriptsCan run jobs in parallelSeveral other plugins do the same
![Page 50: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/50.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
![Page 51: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/51.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
..
Pipelines
.
build ( "syntax" )build ( "lint" )build ( "rspec" )build ( "doc" )build ( "package" )build ( "release" )
![Page 52: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/52.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
..
Parallel jobs
.
parallel ({
build ( "syntax" )},{
build ( "lint" )},{
build ( "rspec" )},
)
![Page 53: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/53.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
![Page 54: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/54.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
..
Conditionals
.
if ( params["RELEASE"] == "true" ) {switch ( params["DEPLOY_METHOD"] ) {
case "librarian":build ( "librarian" )
case "r10k":build ( "r10k" )
case "package":build ( "package" )
}}
![Page 55: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/55.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
![Page 56: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/56.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
..
Build parameters
.
build ( "gitsubmodules",NAME: params["NAME"],ENVIRONMENT: params["ENVIRONMENT"]
)
![Page 57: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/57.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow: Groovy scripts
![Page 58: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/58.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
.. But also…
A ”retry” function to try the failing jobs 2, 3, 4 timeA ”rescue” function to launch a jobs if another failsThe first job fails if one of the pipeline jobs fails!And a lot of other stuffs
![Page 59: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/59.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Plugin #2: Job Generator
Jenkins jobs generatorCreates jobs on the flyJobs do not need to be created at the start of the (flow)pipelineAlso updates existing jobsAllows to keep jobs per project (per project history andgraphes)Takes special parameters
![Page 60: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/60.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Build Flow Plugin + Job generator
One starting job (orchestrator)Calls some jobs or some generators, in parallelEnds with a nice, up-to-date Pipeline
![Page 61: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/61.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Plugins/How
.
..Plugin #3: JobConfigHistory Plugin
Keeps track of the changes in the jobsAlternative to keep the xml in git
![Page 62: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/62.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
Conclusion
![Page 63: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/63.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server
: Think slavesSCM pulling: Think Gerrit or TriggerJobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 64: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/64.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slaves
SCM pulling: Think Gerrit or TriggerJobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 65: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/65.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slavesSCM pulling
: Think Gerrit or TriggerJobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 66: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/66.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slavesSCM pulling: Think Gerrit or Trigger
Jobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 67: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/67.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slavesSCM pulling: Think Gerrit or TriggerJobs are slow
: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 68: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/68.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slavesSCM pulling: Think Gerrit or TriggerJobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications
: Think XMPP, IRC, mails, Gerrit…
![Page 69: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/69.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Other bottlenecks/problems at scale
Load of the Jenkins server: Think slavesSCM pulling: Think Gerrit or TriggerJobs are slow: Optimize, Divide them, Monitor them(graphite)Notifications: Think XMPP, IRC, mails, Gerrit…
![Page 70: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/70.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
..Pipelines at scale…
Two different approachesapplication-levelinfrastructure-level
Pick the one you prefer (or invent your own)Do not hesitate to separate JobsMonitor your jobs/pipelines (graphite)
![Page 71: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/71.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
Thank you!
Thanks to my colleague @tomdevylder for the Build Flow slides
![Page 72: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/72.jpg)
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.Conclusion
.
.. Contact
Julien [email protected]@roidelapluie
Inuits BVBABelgium+32 473 441 636https://inuits.eu
![Page 73: Building Jenkins Pipelines at Scale](https://reader034.vdocuments.mx/reader034/viewer/2022042613/554bc1b0b4c90530298b517e/html5/thumbnails/73.jpg)
Jenkins User Conference Europe #jenkinsconf
Corporate Community
Thank You To Our Sponsors
Platinum Gold
Silver
.
.
.
Jenkins User Conference Europe
.
#jenkinsconf
.#jenkinsconf