symfonycon berlin 2016 jenkins deployment pipelines

1Nicole Cordes, SymfonyCon Berlin 2016, Jenkins Deployment Pipelines

Jenkins Deployment Pipelines

» Introduction

» Jenkins

» Pipelines

» Hands on

» Improvements

» Downsides


Introduction


Me, myself and I

» Nicole Cordes

» working at CPS-IT GmbH in Berlin

» TYPO3 community member since 2011

» Core and Security Team member

» contributed some Symfony improvements for Windows in Finder

» some libraries and own projects based on Symfony

» Twitter: @IchHabRecht

» GitHub: https://github.com/IchHabRecht


About this talk

» who or what is Jenkins

» understanding pipelines

» my way of pipeline integration into Jenkins

» useful Jenkins plugins to be more productive

» drawbacks of the solution


Jenkins


Jenkins

» automation (continuous integration) server

» open source project

» written in Java

» previously known as Hudson

» 2011 Oracle bought Sun and wanted to develop a commercial version

» the development was renamed to Jenkins and continued by the Hudson inventors


Jenkins

» easy to install / maintain

» able to run any kind of customized task

» you can define multiple triggers

» expansible by a huge amount of different plugins

» distribute work across multiple machines


Pipelines


Pipelines

» pipeline means multiple stages with different tasks to do

» build stage

» test stage

» deploy stage

» tool stage


Job pool

tool

SonarQube

phpDocumentor

test

PHPUnit

JUnit

Behat

Codeception

PHP Lint

build

composer

Node.js

asset management

deploy

rsync (any tool)

clear cache

warmup cache

database migrations


Understanding workspaces

» build and test stage share a workspace

» build stage ensures a proper state of your project

» test stage runs tests and ensures stability

» deployment has its own workspace

» you don’t want to deploy all dev tools

» tool stage usually uses deployment workspace as well but can have its own one


Build your project

test

PHPUnit

PHP Lint

build

composer

deploy

rsync (any tool)

» Project A (simple project)

test

PHPUnit

PHP Lint

» Project B (advanced project)

build

composer

Node.js

deploy

Deployer

clear cache

tool

SonarQube

phpDocumentor


Job pool

» each task is an own Jenkins job

» you can easily set up new projects from the job pool

» you can add new jobs to the job pool

» you can change one job and it is changed in every project


Hands on(Project A)


Installing Plugins

» three required plugins to support minimal pipelines

» Git plugin

» Environment Injector Plugin

» Parameterized Trigger plugin

» Copy Artifact Plugin


Create the first job

» enter template name

» choose to build a “Freestyle project”

» allow template jobs to use artifacts

» choose “Prepare an environment for the run”

» enter some project server configuration

» those properties will be available as ENV vars



» define your “Source Code Management”

» choose “Git” and enter your information

» enable “Poll SCM” trigger without any schedule

» this allows you to call

http://jenkins/git/notifyCommit?url=https://github.com/IchHabRecht/some-project.git

to trigger your build from any other resource



» add a build step “Execute shell”

» create a tar file of the Git checkout

» add a post-build action “Archive the artifacts”

» enter your tar file name

» activate “Fingerprint all archived artifacts“


Create the template jobs

» create a new job “template-build-simple-project” as Freestyle project

» activate “Advanced Project Options“

» use a custom workspace

“${ROOT_JOB_NAME}_build/workspace“

» add a build step “Copy artifacts from another project”

» use project name “${ROOT_JOB_NAME}”

» use specific build “${ROOT_BUILD_NUMBER}”


Create the template jobs

» add a build step “Execute shell”

» untar the artifact and delete it

» now our build workspace is prepared to run single tasks


Create the template tasks

» create a new job “template-build-composer-install” as Freestyle project

» use custom workspace “${TEMPLATE_BUILD_WORKSPACE}”

» add a build step “Execute shell” with command `composer install`

» create a new job “template-test-phpunit” as Freestyle project


» add a build step “Execute shell” with command `bin/phpunit`

» depending on your framework you might need more configuration here


Create the template tasks

» create a new job “template-test-php-lint” as Freestyle project


» add a build step “Execute shell” with command

for file in $(find “${WORKSPACE}/” -name \*.php); do

php -d display_errors=stderr -l “$file” > /dev/null

done

» create a new job “template-deploy-rsync” as Freestyle project

» use custom workspace “${TEMPLATE_DEPLOY_WORKSPACE}”

» add a build step “Execute shell” with some rsync function

rsync -rvzle ssh -E --delete “${WORKSPACE}/” “user@${SERVER}:$

{WEB_ROOT}/”


Create project type templates

» create a new job “template-test-simple-project” as Freestyle project

» no specific settings needed so far as tests using the build workspace

» create a new job “template-deploy-simple-project” as Freestyle project

» activate “Advanced Project Options“

» use a custom workspace

“${ROOT_JOB_NAME}_deploy/workspace“

» add a build step “Copy artifacts from another project”

» use project name “${ROOT_JOB_NAME}”

» use specific build “${ROOT_BUILD_NUMBER}”

» add a build step “Execute shell” and untar the artifact and delete it


Template view


Connecting the deploy jobs

» re-configure “template-deploy-simple-project”

» add a new build step “Trigger/call builds on other projects”

» projects to build “template-deploy-rsync”

» enable “Block until the triggered projects finish their builds”

» add parameters “Current build parameters”

» add parameters “Predefined parameters”

TEMPLATE_DEPLOY_WORKSPACE=${WORKSPACE}


Connecting the deploy jobs


Connecting the test jobs

» re-configure “template-test-simple-project”


» projects to build “template-test-phpunit,template-test-php-lint”




Connecting the build jobs

» re-configure “template-build-simple-project”


» make sure it is the last step in this job

» projects to build “template-build-composer-install”




TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}


Connecting the build jobs

» add another new build step “Trigger/call builds on other projects”


» projects to build “template-test-simple-project”




TEMPLATE_BUILD_WORKSPACE=${WORKSPACE}

» add a post-build action “Trigger parameterized build on other projects”

» projects to build “template-deploy-simple-project”




Adding the pipeline to Project A

» re-configure “Project A”

» add a post-build action “Trigger parameterized build on other projects”


» projects to build “template-build-simple-project”




ROOT_JOB_NAME=${JOB_NAME}

ROOT_BUILD_NUMBER=${BUILD_NUMBER}


Conclusions

» you can now build Project A which will trigger all other jobs

» currently only the checkout is deployed; you maybe have to add a

“template-deploy-composer-install” job running `composer install –no-dev`

» you can add a new “Project C” as another simple project

» create a new job and enter a name

» Mark “Copy existing Item” and choose “Project A”

» you just need to change the server variables and the git repository

» you are able to extend your job pool and define other project templates


Improvements


SCM Sync Configuration Plugin

» store your Jenkins configuration in a Git repository

» comment your changes

» bulk edit jobs


Green Balls

» turn the blue status icons into green ones


Role-based Authorization Strategy

» manage and assign roles to users

» global roles like “Administrator” “Developer” “Guest”

» project roles based on project template names

» use client names as project prefixes


Workspace Cleanup Plugin

» delete the workspace before a new build is started

» useful for the “umbrella” template jobs


Email Extension Plugin

» versatile email notification plugin

» send emails to all commiters

» send emails to project members

» attach build logs


Further template improvements

» variable PHP version per project

» Node.js caching

» multiple server support for deployment

» multiple server types according to Git Flow branching model


Downsides


Missing features

» due to the post-build actions the root job never fails

» job pool tasks don’t save state per root template nor Git branch


Thank youfor your attention!

symfonycon berlin 2016 jenkins deployment pipelines

Internet