©2013 cloudbees, inc. all rights reserved 1 jenkins scalability summit oct 2013
TRANSCRIPT
![Page 1: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/1.jpg)
©2013 CloudBees, Inc. All Rights Reserved 1©2013 CloudBees, Inc. All Rights Reserved
Jenkins Scalability SummitOct 2013
![Page 2: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/2.jpg)
©2013 CloudBees, Inc. All Rights Reserved 2
Logistics
• Where is …?• Network
![Page 3: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/3.jpg)
©2013 CloudBees, Inc. All Rights Reserved 3
Agenda Part 1: Story time
• Learn from our collective experience• Identify high-priority problems• What is hurting serious users?• Concrete details are good
![Page 4: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/4.jpg)
©2013 CloudBees, Inc. All Rights Reserved 4
Agenda Part 2: Discuss & Design
• Collectively pick a few topics• Split into 2 tracks
• See if we can start shaping up solutions
• Please keep shared notes– Details!
![Page 5: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/5.jpg)
©2013 CloudBees, Inc. All Rights Reserved 5
Possible topics includes …
• CPU/memory/disk/network/… consumption• Stability/diagnosability of slaves• Access control of builds/UI• Organizing jobs and build records• Master to master communication• Workflow / choreography• User interface• Stability of Jenkins releases• Plugin compatibilities
![Page 6: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/6.jpg)
©2013 CloudBees, Inc. All Rights Reserved 6
Goals
• No one goes listen-only mode• Produce notes to show to the broader
community• Figure out how to do this better next
year
![Page 7: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/7.jpg)
©2013 CloudBees, Inc. All Rights Reserved 7©2013 CloudBees, Inc. All Rights Reserved
What We’ve Done&
What We Can
![Page 8: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/8.jpg)
©2013 CloudBees, Inc. All Rights Reserved 8
Thread Reduction
• Per-slave consumption– SSH (down to 1)– Channel (down to 1)
• Executor thread on demand
![Page 9: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/9.jpg)
©2013 CloudBees, Inc. All Rights Reserved 9
Thread Reduction: TODO
• NIO• Asynchronous job execution–More about this in workflow
![Page 10: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/10.jpg)
©2013 CloudBees, Inc. All Rights Reserved 10
Memory reduction
• Lazy loading of build records• Database plugin
![Page 11: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/11.jpg)
©2013 CloudBees, Inc. All Rights Reserved 11
Design Choices That Constrain Us
• Compatibility– Adding is doable, changing is often hard
• Heterogeneousness in data model• Thread-driven execution model
![Page 12: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/12.jpg)
©2013 CloudBees, Inc. All Rights Reserved 12©2013 CloudBees, Inc. All Rights Reserved
CloudBees Scalability Efforts
![Page 13: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/13.jpg)
©2013 CloudBees, Inc. All Rights Reserved 13
Multi-master / Meta Jenkins
• Master-to-master channel• Extensions on top of it– “Cloud” impl to lease slaves– Push security realm– Push update center–…
![Page 14: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/14.jpg)
©2013 CloudBees, Inc. All Rights Reserved 14
Multi-master
• Where we think we can go with this–Move jobs around– Sharding with UI mashup
• Where this wouldn’t take us– Loss of inflight builds– True horizontal scaling
![Page 15: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/15.jpg)
©2013 CloudBees, Inc. All Rights Reserved 15
Workflow
• Brand new job type• Kill multiple birds in one stone• Inspired by buildflow & jenkow
![Page 16: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/16.jpg)
©2013 CloudBees, Inc. All Rights Reserved 16
Ingredients
• No single thread to execute the whole thing– Analogous to NIO– Check pointing
• BPMN-like workflow execution model– Surface syntax independent
• Groovy DSL–With continuation-passing style execution
model
• Post-execution visualization– But no pre-execution visualization
![Page 17: ©2013 CloudBees, Inc. All Rights Reserved 1 Jenkins Scalability Summit Oct 2013](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cef5503460f949bda1d/html5/thumbnails/17.jpg)
©2013 CloudBees, Inc. All Rights Reserved 17
Ingredients
• Unified properties, build variables, and environments
• Open up direct access to file store• Hands-free process forking• Interop with existing job types