version your build process as you version your code

Version your build process as you version your codeVincent Latombe

@Vlatombe

Footer

#jenkinsconf

The speaker

Software Engineer at CloudBees

Heavy Jenkins user since 2010 Maintainer of the Clearcase plugin Contributes to core, git and literate

2

Footer

#jenkinsconf

Agenda

What is Literate ? Literate in Practice Literate in real life Under the hood

3

#jenkinsconf

What is Literate?

Footer

#jenkinsconf

Literate Programming

• Introduced by Donald Knuth in 1983

• Explain program logic using natural language with snippets of code and macros

5

Footer

#jenkinsconf

Literate Builds

• Applies Literate paradigm to builds • Describes the build steps in a marker file

or simply in README.md • Build definition stored in the same SCM

as the code it is building

6

Footer

#jenkinsconf

Literate Builds

• Marker file within the SCM • May contain the build

definition • Fallback to README.md if

empty

7

Footer

#jenkinsconf

Markdown Cheatsheet

A section ========= This is text inside section. And `this is inline code`.

some code

8

Footer

#jenkinsconf

Hello world example

Hello world literate project ============================

Build -----

Let's say hello

echo "Hello world"

9

Footer

#jenkinsconf

Ant example

Environments =========== Notice how we specify the environment to build with by providing Jenkins node labels or tool installer names in code snippet sections attached to bullet points in an "environment" section?

* `ant-1.8`, `java-1.7`

Build =====

ant clean dist

10

Footer

#jenkinsconf

Matrix build? No problem

Complex project ===============

Environments ===========

We have two different environments that the build must be run on:

* `linux`, `gcc-4.2`, `ant-1.8`, `maven-3.0.5`, `java-1.7` * `windows`, `vs-pro-2012`, `ant-1.8`, `maven-3.0.5`, `java-1.7`

11

Footer

#jenkinsconf

Build commands per environment

Build =====

We have two different sets of build instructions, one for building with visual studio and the other for building with GCC

* On `gcc-4.2`, we start by building the native code

./configure make

* On `vs-pro-2012`, we have a batch file to do the native steps

call build-native.bat

12

Footer

#jenkinsconf

Additional tasks

• Defined in the marker file • Enabled per branch using the job configuration • Lightweight promotion

– Self-promotion – Manual promotion (with optional parameters)

13

Footer

#jenkinsconf

Extensions (1/2) : job configuration

• Things you want as a Jenkins admin – Timestamp logs – Discard old builds – Clean up workspace before build – Notification to GitHub/Stash

– Use Branch Properties

14

Footer

#jenkinsconf

Extensions (2/2) : in SCM

• Things you leave up to the developer – Files stored in SCM under .jenkins folder – Basic: <extensionName>.xml, contains the config snippet

found in config.xml – Implement Agent to expose higher-level configuration files

– location of test results – coverage threshold

15

#jenkinsconf

Literate in practice

Footer

#jenkinsconf

Installation

• Set up experimental update center • http://updates.jenkins-ci.org/experimental/update-center.json • Check for updates

• Install Literate plugin and dependencies • Restart if needed

17

Footer

#jenkinsconf

DEMO TIME !

• https://github.com/Vlatombe/literate-sample

18

#jenkinsconf

Literate in real life

Footer

#jenkinsconf

Context

• About 3000 developers • Commercial products

– ~1000 Java/JEE – ~2000 C++

• For internal tooling, add – Python – Ruby – Perl

20

Footer

#jenkinsconf

It’s about control!

• Fully open • Locked down • Balanced

– Template-based solutions: only defined attributes can be customized – Literate: commands can be customized, post-build actions can be whitelisted

21

Footer

#jenkinsconf

Infrastructure

22

Stash Jenkins

❑ 1.Enable CI

2.Creates

3.Index branches

4.Parses marker

Literate Multi-branch project

Literate branch project

Project/Repository

Literate env. project

Literate env. project

5.Update build status

Footer

#jenkinsconf

YAML Parser

• Markdown considered too textual, too much space for syntax errors, even if using README.md was tempting

• YAML fanboys • New parser was easy to implement thanks to existing Literate

architecture

23

Footer

#jenkinsconf

Markdown vs YAML

Environments ============

* `windows`, `java-1.6` * `linux`, `java-1.7` Build =====

mvn –B clean verify

24

environments: - [windows, java-1.6] - [linux, java-1.7] build: mvn –B clean verify

YAMLMarkdown

versus

#jenkinsconf

Under the hood

Footer

#jenkinsconf

Architecture

26

Literate-plugin

Literate-apiBranch-api

Scm-api

Footer

#jenkinsconf

Literate-api

• Builds a Project Model from a text file • Doesn’t depend on Jenkins • Pluggable model builders

– Markdown – YAML

27

Footer

#jenkinsconf

Scm-api

• jenkins.scm.api.SCMSource : provider for hudson.scm.SCM instances

• Able to comprehend multiple heads on a SCM

– Git branches – Github pull requests – Gerrit reviews

28

Footer

#jenkinsconf

Branch-api

• Toolkit to handle multi-branch – Dead branch policy – Untrusted branches – Build retention – Throttling

• Foundation for – Multi-branch freestyle project – Multi-branch template project – Multi-branch workflow (see demo by @tyvole at 1pm)

29

Footer

#jenkinsconf

Literate-plugin

• Extension points – LiterateBranchProperty : decorate branch and environments with anything

(buildwrapper, properties…) – Agent : Builds Publisher instances from files in the SCM.

30

Footer

#jenkinsconf

What’s next?

• Integration with multi-branch workflow – Use the literate descriptor to feed a workflow – Branch sources for GitHub, Bitbucket, Stash pull requests – Additional plugins compatibility

• Essentially the same problem as with workflow, without the necessary investment behind

31

Footer

#jenkinsconf

Takeout

• Drive your build process from your SCM • Use a description markup, independant from Jenkins • Proper multi-branch support!

• Give it a try !

32

#jenkinsconf

Footer

Thanks to our Sponsors!

33