![Page 1: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/1.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Best Practices for a Mission-Critical Jenkins
Mike Rooney Jenkins Connoisseur
http://linkedin.com/in/mcrooney
![Page 2: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/2.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Jenkins Uses
! Genius.com – staging deployment, code reviews,
automated branching and merging, monitors
! Canv.as – continuous deployment, scoring, monitoring,
newsletter mailing
! Conductor – environment creation, staging / prod
deployment, selenium monitoring
![Page 3: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/3.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Hand-check: How critical is your Jenkins?
![Page 4: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/4.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
What problems have you faced?
![Page 5: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/5.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Problems
! disk failure / data loss ! hardware failure / downtime ! load / latency
![Page 6: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/6.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Solution
! make Jenkins instance trivial to respin – ideally a one-liner that even handles DNS – “create.sh jenkins”
![Page 7: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/7.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Persistence
! $JENKINS_HOME – plugins, users, jobs, builds, configuration
![Page 8: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/8.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Persistence
! git / svn – make $JENKINS_HOME a checkout – have a Jenkins job that commits daily – examples: http://jenkins-ci.org/content/
keeping-your-configuration-and-data-subversion
![Page 9: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/9.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Persistence
! EBS on AWS – put $JENKINS_HOME on an EBS volume – snapshot nightly via a Jenkins job
![Page 10: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/10.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Environment
! Jenkins is more than a .war – specific Jenkins version – startup options – dependent packages: git, ruby gems, pip – ssh keys, m2 settings – swap, tmpfs, system configuration
![Page 11: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/11.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Environment
! configuration management:Puppet/Chef*
* https://wiki.jenkins-ci.org/display/JENKINS/Puppet
![Page 12: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/12.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Environment
! standalone – puppet apply path/to/your/manifest.pp
! puppetmaster – set up /etc/puppet.conf, run puppet agent
![Page 13: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/13.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Putting it Together
! have manifest handle $JENKINS_HOME – clone git repo, mount EBS volume, etc
![Page 14: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/14.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Putting it Together…on AWS
! upload manifests to S3 on check-in – a Jenkins SCM job using S3 plugin
! use cloud-init to install puppet, download manifests, and run puppet – a custom AMI with an rc.local script also works
! when it dies: “create.sh jenkins” – ec2-launch-instance config user-data
![Page 15: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/15.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Monitoring
! … but how do you know when it’s down? ! check out services like Pingdom
– notifies you when a URL does give HTTP 200 OK
![Page 16: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/16.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
! handles provisioning simply from a .war ! pros
– just give it a war – automatically replaces unhealthy instances – behind a load-balancer (consistent URL) – normally hard AWS changes like AMI, Security
Groups, or Key Pairs are now trivial to make
! cons – behind a load-balancer (cost overhead) – no UI option (yet) for controlling AZ – no great way to pass data to instances for puppet – locked in to Amazon Linux AMI (CentOS)
![Page 17: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/17.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
! set min/max instances to 1 – ignore scaling triggers, irrelevant in this case
! use beanstalk CLI to set desired AZ (if EBS) – https://forums.aws.amazon.com/thread.jspa?
threadID=61409
! puppet – use a custom AMI that specifically runs Jenkins
manifests – but this requires a specific AMI for each
Beanstalk application. – let’s get creative…
![Page 18: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/18.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Going further: Elastic Beanstalk
! passing data to instances
! PARAM1..5 meant as args to .war ! end up in /etc/sysconfig/tomcat7 JAVA_OPTS ! parse out and:
– puppet apply –certname=$PARSED_ROLE
![Page 19: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/19.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Questions?
![Page 20: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/20.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
High Availability Artifacts
! protect: artifacts, reports, userContent ! from:
– planned downtime: Jenkins restarts/upgrades, server upgrades
– unplanned downtime: software/hardware failure
![Page 21: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/21.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
High(er) Availability Artifacts
! easy mode: – put Jenkins behind nginx/apache, shadow
userContent and relevant directories – still available during Jenkins restarts, or very
high Jenkins load/latency – not safe from server downtime
![Page 22: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/22.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
High Availability Artifacts
! advanced mode: S3 – 99.99% availability, 99.999999999% durability*
• if you store 10K objects, expect to lose one every 10 million years
– use Jenkins S3 plugin to upload artifacts to S3
* http://aws.amazon.com/s3/faqs
![Page 23: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/23.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Fault-tolerant Jobs
! design with possible downtime in mind – SCM triggering is great, but keep polling too
![Page 24: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/24.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Fault-tolerant Jobs
! */15 * * * – BAD:
update users where join_time < 15m ago – GOOD:
update users where id > last_id_updated
![Page 25: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/25.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Error handling
! for non-critical jobs, use email / IM post-build notifiers – but be careful of creating too much noise,
people will ignore or filter it out
! for critical jobs, integrate Jenkins with a service like PagerDuty – Jenkins emails [email protected] – PagerDuty texts / calls the people on-call until
resolved – a failing build will wake you up at 4AM
![Page 26: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/26.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Questions?
![Page 27: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/27.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: Authentication
! read-only ! matrix-based ! HTTP basic auth
![Page 28: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/28.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: Authentication
! but what about traffic sniffing?
![Page 29: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/29.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: HTTPS
! throw nginx/apache in front of Jenkins #EXAMPLE
![Page 30: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/30.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: Authorization
! use project-based matrix authentication ! give anonymous/authenticated readonly ! use it if you’ve got it:
LDAP, Active Directory, UNIX ! Jenkin’s own database also works fine ! ensure each user has their own account
![Page 31: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/31.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: Authorization (AWS)
! when interfacing with AWS API/CLI, use IAM so Jenkins can only access what it needs
![Page 32: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/32.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Security: Audit Trails
![Page 33: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/33.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Questions?
![Page 34: Best Practices for a Mission- Critical JenkinsBest Practices for a Mission-Critical Jenkins Mike Rooney Jenkins Connoisseur ... Genius.com – staging deployment, code reviews, automated](https://reader034.vdocuments.mx/reader034/viewer/2022050314/5f772135eb2ab27f943554d7/html5/thumbnails/34.jpg)
Jenkins User Conference New York, May 17 2012 #jenkinsconf
Thank You To Our Sponsors Platinum Sponsor
Gold Sponsors
Silver Sponsors
Bronze Sponsors