symfonycon berlin 2016 jenkins deployment pipelines

1Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Jenkins Deployment Pipelines

» Introduction

» Jenkins

» Pipelines

» Hands on

» Improvements

» Downsides

2Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Introduction

3Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Me, myself and I

» Nicole Cordes

» working at CPS-IT GmbH in Berlin

» TYPO3 community member since 2011

» Core and Security Team member

» contributed some Symfony improvements for Windows in Finder

» some libraries and own projects based on Symfony

» Twitter: @IchHabRecht

» GitHub: https://github.com/IchHabRecht

4Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

About this talk

» who or what is Jenkins

» understanding pipelines

» my way of pipeline integration into Jenkins

» useful Jenkins plugins to be more productive

» drawbacks of the solution

5Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Jenkins

6Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Jenkins

» automation (continuous integration) server

» open source project

» written in Java

» previously known as Hudson

» 2011 Oracle bought Sun and wanted to develop a commercial version

» the development was renamed to Jenkins and continued by the Hudson inventors

7Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Jenkins

» easy to install / maintain

» able to run any kind of customized task

» you can define multiple triggers

» expansible by a huge amount of different plugins

» distribute work across multiple machines

8Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Pipelines

9Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Pipelines

» pipeline means multiple stages with different tasks to do

» build stage

» test stage

» deploy stage

» tool stage

10Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Job pool

tool

SonarQube

phpDocumentor

test

PHPUnit

JUnit

Behat

Codeception

PHP Lint

build

composer

Node.js

asset management

deploy

rsync (any tool)

clear cache

warmup cache

database migrations

11Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Understanding workspaces

» build and test stage share a workspace

» build stage ensures a proper state of your project

» test stage runs tests and ensures stability

» deployment has its own workspace

» you don’t want to deploy all dev tools

» tool stage usually uses deployment workspace as well but can have its own one

12Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Build your project

test

PHPUnit

PHP Lint

build

composer

deploy

rsync (any tool)

» Project A (simple project)

test

PHPUnit

PHP Lint

» Project B (advanced project)

build

composer

Node.js

deploy

Deployer

clear cache

tool

SonarQube

phpDocumentor

13Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Job pool

» each task is an own Jenkins job

» you can easily set up new projects from the job pool

» you can add new jobs to the job pool

» you can change one job and it is changed in every project

14Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Hands on(Project A)

15Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Installing Plugins

» three required plugins to support minimal pipelines

» Git plugin

» Environment Injector Plugin

» Parameterized Trigger plugin

» Copy Artifact Plugin

16Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the first job

» enter template name

» choose to build a “Freestyle project”

» allow template jobs to use artifacts

» choose “Prepare an environment for the run”

» enter some project server configuration

» those properties will be available as ENV vars

17Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the first job

» define your “Source Code Management”

» choose “Git” and enter your information

» enable “Poll SCM” trigger without any schedule

» this allows you to call

http://jenkins/git/notifyCommit?url=https://github.com/IchHabRecht/some-project.git

to trigger your build from any other resource

18Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the first job

» add a build step “Execute shell”

» create a tar file of the Git checkout

» add a post-build action “Archive the artifacts”

» enter your tar file name

» activate “Fingerprint all archived artifacts“

19Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the template jobs

» create a new job “template-build-simple-project” as Freestyle project

» activate “Advanced Project Options“

» use a custom workspace

“${ROOT_JOB_NAME}_build/workspace“

» add a build step “Copy artifacts from another project”

» use project name “${ROOT_JOB_NAME}”

» use specific build “${ROOT_BUILD_NUMBER}”

20Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the template jobs

» add a build step “Execute shell”

» untar the artifact and delete it

» now our build workspace is prepared to run single tasks

21Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the template tasks

» create a new job “template-build-composer-install” as Freestyle project

» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”

» add a build step “Execute shell” with command `composer install`

» create a new job “template-test-phpunit” as Freestyle project

» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”

» add a build step “Execute shell” with command `bin/phpunit`

» depending on your framework you might need more configuration here

22Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create the template tasks

» create a new job “template-test-php-lint” as Freestyle project

» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”

» add a build step “Execute shell” with command

for file in $(find “${WORKSPACE}/” -name \*.php); do

php -d display_errors=stderr -l “$file” > /dev/null

done

» create a new job “template-deploy-rsync” as Freestyle project

» use custom workspace “${TEMPLATE_DEPLOY_WORKSPACE}”

» add a build step “Execute shell” with some rsync function

rsync -rvzle ssh -E --delete “${WORKSPACE}/” “user@${SERVER}:$

{WEB_ROOT}/”

23Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Create project type templates

» create a new job “template-test-simple-project” as Freestyle project

» no specific settings needed so far as tests using the build workspace

» create a new job “template-deploy-simple-project” as Freestyle project

» activate “Advanced Project Options“

» use a custom workspace

“${ROOT_JOB_NAME}_deploy/workspace“

» add a build step “Copy artifacts from another project”

» use project name “${ROOT_JOB_NAME}”

» use specific build “${ROOT_BUILD_NUMBER}”

» add a build step “Execute shell” and untar the artifact and delete it

24Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Template view

25Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Connecting the deploy jobs

» re-configure “template-deploy-simple-project”

» add a new build step “Trigger/call builds on other projects”

» projects to build “template-deploy-rsync”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

» add parameters “Predefined parameters”

TEMPLATE_DEPLOY_WORKSPACE=${WORKSPACE}

26Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Connecting the deploy jobs

27Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Connecting the test jobs

» re-configure “template-test-simple-project”

» add a new build step “Trigger/call builds on other projects”

» projects to build “template-test-phpunit,template-test-php-lint”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

28Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Connecting the build jobs

» re-configure “template-build-simple-project”

» add a new build step “Trigger/call builds on other projects”

» make sure it is the last step in this job

» projects to build “template-build-composer-install”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

» add parameters “Predefined parameters”

TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}

29Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Connecting the build jobs

» add another new build step “Trigger/call builds on other projects”

» make sure it is the last step in this job

» projects to build “template-test-simple-project”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

» add parameters “Predefined parameters”

TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}

» add a post-build action “Trigger parameterized build on other projects”

» projects to build “template-deploy-simple-project”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

30Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Adding the pipeline to Project A

» re-configure “Project A”

» add a post-build action “Trigger parameterized build on other projects”

» make sure it is the last step in this job

» projects to build “template-build-simple-project”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

» add parameters “Predefined parameters”

ROOT_JOB_NAME=${JOB_NAME}

ROOT_BUILD_NUMBER=${BUILD_NUMBER}

31Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Conclusions

» you can now build Project A which will trigger all other jobs

» currently only the checkout is deployed; you maybe have to add a

“template-deploy-composer-install” job running `composer install –no-dev`

» you can add a new “Project C” as another simple project

» create a new job and enter a name

» Mark “Copy existing Item” and choose “Project A”

» you just need to change the server variables and the git repository

» you are able to extend your job pool and define other project templates

32Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Improvements

33Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

SCM Sync Configuration Plugin

» store your Jenkins configuration in a Git repository

» comment your changes

» bulk edit jobs

34Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Green Balls

» turn the blue status icons into green ones

35Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Role-based Authorization Strategy

» manage and assign roles to users

» global roles like “Administrator” “Developer” “Guest”

» project roles based on project template names

» use client names as project prefixes

36Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Workspace Cleanup Plugin

» delete the workspace before a new build is started

» useful for the “umbrella” template jobs

37Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Email Extension Plugin

» versatile email notification plugin

» send emails to all commiters

» send emails to project members

» attach build logs

38Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Further template improvements

» variable PHP version per project

» Node.js caching

» multiple server support for deployment

» multiple server types according to Git Flow branching model

39Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Downsides

40Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Missing features

» due to the post-build actions the root job never fails

» job pool tasks don’t save state per root template nor Git branch

41Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Thank youfor your attention!