Scaling Jenkins with Kubernetes 1.2

About meSenior DevOps Engineer @ Gliderubygems.org/profiles/amimahloofContribution to OpenSource via RubyGems and GitHub:

github.com/innovia

linkedin.com/in/amimahloof

Glide environment

Multiple Redis ServersMySQLDynamoDBMultiple Background Queue ProcessorsApp ServerMultiple Web Socket ServersCloudSearch

Previous Jenkins setup

Single Jenkins Master machine for Server code builds

Single Jenkins Master Machine for Android client builds

Dedicated MySQL database per build

Dedicated port per build service

Each build environment encapsulated and managed by Eye - process monitoringhttps://github.com/kostya/eye

Previous Jenkins IssuesUnder-utilized instances running 24/7 (expensive!)Port collisionsOut of memory issues - build failuresDatabases needed to be created and wiped per buildDebugging failed tests was extremely hardUpdating Jenkins ec2 image especially for Android was a challenge

NFS-SERVERPOD

EBS VolumeScaling Jenkins with Kubernetes

Jenkins Master BackEnd RC

Jenkins Master Android RC

Backend POD

Jenkins Ingress ControllerPOD

nginx configmap

Jenkins SlavePODWith KubeCtl

Backend Service (IP)

single elb for both CIs addresses

Current Jenkins Kubernetes Pluginhttps://github.com/jenkinsci/kubernetes-pluginSupport single docker image(jenkins-slave docker image - it creates a POD with that image)

Does not support multiple containers in a POD

Does not support Persistent Volumes

Modified Jenkins Kubernetes Pluginhttps://github.com/innovia/kubernetes-pluginSupport reading and parsing a POD template file

Support multiple containers in a POD

Support Persistant Volumes, EBS, EmptyDir, HostPath Volumes

Does not support POD template per JOB (System global config)

Read POD TemplateStart Build

Send POD Object

Jenkins SlavePOD

download JNLP file from Jenkins Master

Connect to Jenkins Master Slave Port

Build Flow Part 1

Pull Request

Build Flow Part 2

Jenkins Master

Jenkins SlaveStart Job on Slave

git fetch and merge server code

create serviceJOB_NAME-BUILD_NUMBER

create PV - PVCJOB_NAME-BUILD_NUMBER

start tests

teardown POD, SERVICE, PV, PVC

clean-up temp location

submit result to bitbucket

create PODJOB_NAME-BUILD_NUMBER

wait for setup complete file (check dependancies)

AdvantagesOn Demand PODs Complete test isolationSource controlled by developers flexible POD templateJobs queued up in Kubernetes Scheduler until node resources are available (running builds finish)Scalable Kubernetes Nodes Plugin keeps track of POD - if POD dies in the middle it launches up a new slave for that job Persistant Storage (EBS via NFS)Spot instances using node selectors

Demo

Kubernetes on Spot InstancesSave 70-90% over On DemandManaged by Spot Fleet to reduce downtimeDifferent fleets for different types of Kubernetes Nodes, allowing dynamic pod allocation using Node Selector

Until implemented in Kubernetes 1.3, we have to create our own Fleets

eg. https://github.com/kubernetes/kubernetes/issues/24472#issuecomment-211975112

Thank you!Questions?Slides available http://www.slideshare.net/AmiMahloof/scaling-jenkins-with-kubernetes