using the pipeline plugin to accelerate cd (1)
TRANSCRIPT
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
1/21
Using the Pipeline Pluginto Accelerate ContinuousDelivery
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
2/21
Table of Contents!"#$%&'(#!%") #+, -!-,.!", -.'/!" 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 1203 !"4#5..!"/ #+, -!-,.!", -.'/!" 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 6
703 ($,5#!"/ 5 -!-,.!", 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 6"#$%$&' ()*+ ,$,"-$&" ................................................................................................................................................. /*0$&' %1" 0&$,,"% '"&"+2%)+ ..................................................................................................................................... /-)2#$&' "3%"+&2- ,$,"-$&" 04+$,%0............................................................................................................................... 5
103 8'!.&!"/ 9%'$ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 :41"46$&' )*% 4)#" .................................................................................................................................................... 5+*&&$&' ()*+ ,$,"-$&" ............................................................................................................................................... 72##$&' 0%2'"0 2 0%",0 ............................................................................................................................................ 7'"&"+2- 8*$-# 0%",0 ................................................................................................................................................... 7
04+$,%$&'
................................................................................................................................................................... 9603 !"#,/$5#!"/ 9%'$ #%%.4 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 ;
%))-0 ........................................................................................................................................................................ 9'-)82- :2+$28-"0 ....................................................................................................................................................... 9"3$0%$&' ;)80 .............................................................................................................................................................. ?12-$&' "++)+0 ..................................................................................................................................................... >?
:03 4($!-# 4,('$!#9 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 22
@03 5((,44!"/ =!.,4 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 27
0%201$&' @$-"0 .......................................................................................................................................................... >A2+41$:$&' ................................................................................................................................................................ >A
;03 4(5.!"/ 9%'$ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 2141"46,)$&%0 ............................................................................................................................................................. >B,$,"-$&" %"=,-2%"0 ................................................................................................................................................... >B
?03 #9!"/ !# #%/,#+,$) 45A-., -!-,.!", 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 21
2303 &%(B,$ >!#+ -!-,.!", 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 26
2203 ,C#,"&!"/ -!-,.!", 000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 2
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
3/21
3
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
Introduction: The Pipeline PluginJenkins is a powerful, open source automation tool with an impressive plugin architecture
that helps development teams automate their software lifecycle. Jenkins is used to powermany industry-leading companies' software development pipelines.
Jenkins Pipeline is a powerful, first-class feature for managing complex, multi-step pipelines.
Jenkins Pipeline, a set of open source plugins and integrations, brings the power of Jenkins
and the plugin ecosystem into a scriptable Domain Specific Language (DSL). Best of all, like
Jenkins core, Pipeline is extensible by third-party developers, supporting custom extensions
to the Pipeline DSL and various options for plugin integration.
The Pipeline plugin was formerly known as the Workflow plugin prior to version 1.13 of the
plugin.
Image 1: Pipeline Stage View UI
In this whitepaper, we provide an introduction and step-by-step guide on how to use the
Pipeline plugin. We conclude with a full syntex reference card and a real-world delivery
pipeline example, that is built on the more basic Pipeline snippets provided earlier in the
whitepaper.
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
4/21
4
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
1.0 Installing the Pipeline PluginIt is assumed that you have already installed Jenkins either via the CloudBees Jenkins
Platform or from the Jenkins open source project website. Jenkins Version 1.609.1+ isrequired.
Open Jenkins in your web browser
Navigate to Manage Jenkins > Manage Plugins
Navigate to the Available tab, filter by Pipeline
Select the Pipeline plugin and install
Restart Jenkins
This whitepaper was written using Pipeline version 1.13. Installing the Pipeline plugin installs
all necessary Pipeline dependencies and a new job type called Pipeline.
2.0 Creating a PipelineNow that you have Jenkins running and have installed the Pipeline plugin, you are ready
to create your first pipeline. Create a new pipeline by selecting New Item from the Jenkins
home page.
First, give your pipeline a name, e.g.: "hello-world-flow." Pipelines are simple Groovy scripts,
so let's add the obligatory Hello World. Add a pipeline to the Pipeline script textarea:
!"#$ &'!(($ )$*(+&
Now save your pipeline, ensuring the Use Groovy Sandbox option is checked (more details
to follow on this setting). Click Build Now to run your pipeline.
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
5/21
5
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
Editing Your PipelineBecause pipelines are simple text scripts, they are easy to edit. As you've seen, pipelines can
be edited directly in the Jenkins UI when configuring your pipeline.
Using the Snippet GeneratorTo make editing your pipelines easier, use the Snippet Generator. The Snippet Generator is
dynamically populated with the latest Pipeline steps. Depending on the plugins installed in
your environment, you may see more available steps.
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
6/21
6
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
Loading External Pipeline ScriptsBecause pipelines are text assets, they are ideal to store in a source control system. Pipelines
can be edited in your external IDE then loaded into Jenkins using the Pipeline Script fromSCM option.
3.0 Building Your PipelineNow that you've created a pipeline, let's continue to build on it. For a complex flow, you
should leverage Jenkins' job scheduling queue:
,$+!- .# &/,01!&2
The concept of a node should be familiar to Jenkins users: node is a special step that
schedules the contained steps to run by adding them to Jenkins' build queue. Even better,requesting a node leverages Jenkins' distributed build system. Of course, to select the right
kind of node for your build, the node element takes a label expression:
,$+!3&/,45 66 789:4;&
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
7/21
7
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
Running Your Pipeline
Because pipelines are built as Jenkins jobs, they can be built like other jobs. You can use theBuild Now feature to manually trigger your build on-demand or set up triggers to execute
your pipeline based on certain events.
Adding Stages and StepsStages are usually the top most element of Pipeline syntax. Stages allow you to group your
build step into its component parts. By default, multiple builds of the same pipeline can run
concurrently. The stage element also allows you to control this concurrency:
.;0K! &:/4(+&
,$+!- N 2.;0K! ,01!@ &;!.;&E "$,"/**!,"O@ P
,$+!- N 2
.;0K! ,01!@ &+!L($O&E "$,"/**!,"O@ Q
,$+!- N 2
In this example, we have set a limit of three concurrent executions of the test stage and only
one execution of the deploy stage. You will likely want to control concurrency to prevent
collisions (for example, deployments).
Newer builds are always given priority when entering a throttled stage; older builds will
simply exit early if they are preempted.
General Build StepsWithin your stages, you will add build steps. Just like with Freestyle Jenkins jobs, build steps
make up the core logic of your pipeline. Jenkins Pipeline supports any compatible Build Step
and populates the snippet generator with all available Build Steps in your Jenkins
environment.
.;!L3>?"(0..@ &R0S0+$"T*"#4S!*&E U0S0+$"V4*@ &;0*K!;G*!.$/*"!.GU0S0+$"&E =!!LT((@
J0(.!H?"(0..@ &W4,K!*L*4,;!*&E ;0*K!;.@ &;0*K!;G0L4M)0*&H
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
8/21
8
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
ScriptingJenkins Pipeline supports executing shell (*nix) or batch scripts (Windows) just like freestyle
jobs:
.# &.(!!L QX&
:0; &;41!$/; G; QX&
Scripts can integrate with various other tools and frameworks in your environment - more to
come on tools in the next section.
4.0 Integrating Your ToolsFor a real-life pipeline, Jenkins needs to integrate with other tools, jobs and the underlyingenvironment.
ToolsJenkins has a core capability to integrate with tools. Tools can be added and even
automatically installed on your build nodes. From Pipeline, you can simply use the tool DSL
syntax:
+!J1S,'$1! Y;$$( &DP&
.# Z?-1S,'$1!2G:4,G1S, 9[ S!*4JOZ
In addition to returning the path where the tool is installed, the tool command ensures thenamed tool is installed on the current node.
Global VariablesThe env global variable allows accessing environment variables available on your nodes:
!"#$ !,SM\T]'
Because the env variable is global, changing it directly is discouraged as it changes the
environment globally, so the withEnv syntax is preferred (see example in Full Syntax
Reference Card at the end of this whitepaper).
The currentBuild global variable can retrieve and update the following properties:
"/**!,;[/4(+M*!./(;
"/**!,;[/4(+M+4.L(0O^01!
"/**!,;[/4(+M+!."*4L;4$,
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
9/21
9
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
Existing JobsExisting jobs can be triggered from your pipeline via the build command (e.g.: build 'existing-
freestyle-job'). You can also pass parameters to your external jobs as follows:
+!JU$: Y:/4(+ U$:@ &.0O9#!(($&E L0*01!;!*.@ >>?"(0..@ &B;*4,K\0*01!;!*_0(/!&E ,01!@
&)#$&E S0(/!@ &`#4;!L0L!* I!0+!*.&HH
5.0 Controlling FlowBecause the Pipeline plugin is based on the Groovy language, there are many powerful flow
control mechanisms familiar to developers and operations teams, alike. In addition to
standard Groovy flow control mechanisms like if statements, try/catch and closures there are
several flow control elements specific to Pipeline.
Handling ApprovalsPipeline supports approvals, manual or automated, through the input step:
4,L/; &T*! O$/ ./*!a&
With the submitter parameter, the input step integrates the Jenkins security system to restrict
the allowed approvers.
The input step in Jenkins Pipeline Stage View UI:
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
10/21
10
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
TimingTimeouts allow pipeline creators to set an amount of time to wait before aborting a build:
;41!$/;3;41!@ PXE /,4;@ &BbCc^VB&< - N 2
Parallel stages add a ton of horsepower to Pipeline, allowing simultaneous execution of build
steps on the current node or across multiple nodes, thus increasing build speed:
L0*0((!( &d/0(4;O ."0,&@ -
,$+! -.# &1S, .$,0*@.$,0*&2
2E &4,;!K*0;4$, ;!.;&@ -
,$+! -.# &1S, S!*4JO&2
2
Jenkins can also wait for a specific condition to be true:
)04;e,;4( - N 2
Handling ErrorsJenkins Pipeline has several features for controlling flow by managing error conditions in your
pipeline. Of course, because Pipeline is based on Groovy, standard try/catch semantics apply:
;*O-
2 "0;"#3!< -
2
Pipeline creators can also create error conditions if needed based on custom logic:
4J3f.$/*"!.< -
!**$* &^$ .$/*"!.&
2
Jenkins can also retry specific Pipeline steps if there is variability in the steps for some reason:
*!;*O3g< - N 2
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
11/21
11
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
6.0 Script SecurityAs you've seen, Pipeline is quite powerful. Of course, with power comes risk, so Pipeline has a
robust security and approval framework that integrates with Jenkins core security.
By default, when creating pipelines as a regular user (that is, without the Overall/RunScripts
permission), the Groovy Sandbox is enabled. When the Sandbox is enabled, Pipeline creators
will only be allowed to use pre-approved methods in their flow.
When adding pre-approved methods to a pipeline, script changes do not require approval.When adding a new method (such as a Java API), users will see a RejectedAccessException
and an administrator will be prompted to approve usage of the specific new API or method.
Deselecting the Use Groovy Sandbox option changes this behavior. When the Sandbox
is disabled, pipeline edits require administrator approval. Each change or update by a
non-administrator user requires approval by an administrator. Users will see an
UnapprovedUsageException until their script is approved. Approving individual edits may
not scale well, so the Groovy Sandbox is recommended for larger environments.
-
7/26/2019 Using the Pipeline Plugin to Accelerate CD (1)
12/21
12
USING THE PIPELINE PLUGIN TO ACCELERATE CONTINUOUS DELIVE
7.0 Accessing FilesDuring your pipeline development, you will very likely need to read and write files in your
workspace.
Stashing FilesStashing files between stages is a convenient way to keep files from your workspace to share
them between different nodes:
.;0K! &:/4(+&,$+!-
K4; ;L.@GGK4;#/:M"$1G"($/+:!!.G;$+$90L4MK4;&
.;0.# 4,"(/+!.@ &L$1M51(&E ,01!@ &L$1&
2
.;0K! ,01!@ &;!.;&E "$,"/**!,"O@ P
,$+! -
/,.;0.# &L$1&
.# &"0; L$1M51(&
2
Stash can be used to prevent cloning the same files from source control during different
stages, while also ensuring the same exact files are used during compilation and tested in
later pipeline stages.
ArchivingLike other Jenkins job types, pipelines can archive their artifacts:
0*"#4S! 4,"(/+!.@ &FMU0*&E !5"(/+!.@ &F9.$/*"!.MU0*&
Archives allow you to maintain binaries from your build in Jenkins for easy access later. Unlike
stash, archive keeps artifacts around after a pipeline execution is complete (where stash is
temporary).
Beyond stashing and archiving files, the following Pipeline elements also work with the file
system (more details at the end of this whitepaper):
L)+3