continuous deployment into the unknown with artifactory, bintray, docker and mesos
TRANSCRIPT
![Page 1: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/1.jpg)
© 2015 VMware Inc. All rights reserved.
Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos
Gilad GaronKiril Nesenko
![Page 2: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/2.jpg)
2
Agenda• What is the Common SaaS Platform (CSP)• CI/CD processes for CSP• Upgrading CSP • Xenon - Distributed Control Plane (If we have the time)
![Page 4: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/4.jpg)
4
VMware’s SaaS Transition
• VMware is developing many SaaS offerings • Many services have the same common requirements (Billing, Identity, etc.)• Like other good engineers, we like to reuse code wherever possible• VMware’s Common SaaS Platform (CSP) is platform that internal SaaS
offerings are using to leverage existing internal components
![Page 5: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/5.jpg)
5
Designing a SaaS platform
Design PrinciplesCloud AgnosticHighly Available
ScalableGreat Public APIs
Modular
In PracticeInfrastructure needs to support containersDynamic, Stateful and Distributed clusterTunable consistency helps to achieve availability & scalabilityNo internal APIsCapabilities as libraries, Coupling is done with APIs
Ease of operability / development
Single JAR, limited classpath dependencies set
![Page 6: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/6.jpg)
Deployment Architecture. yep that’s it.
6
Xenon Host Jar
Container
Xenon Host Jar
Container
Xenon Host Jar
Container
Xenon Host Jar
Container
Some Cloud Provider Inc.
![Page 7: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/7.jpg)
Infrastructure and Patch Life Cycle
![Page 8: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/8.jpg)
8
CI/CD Overview
Customer 1 Customer N
Customer 2
automation R&D production
promotion
deploy&test
staging
![Page 9: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/9.jpg)
9
CSP Mesos Infrastructure
![Page 10: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/10.jpg)
10
CI/CD Tools• Artifacts: Artifactory, Bintray• CI: Jenkins• Source Control: git• Code review: gerrit• Slaves: dockers• Infrastructure: mesos, dockers• Code Analysis: Sonar• Build: gradle, Makefiles• Languages: Java, JS, Python, Go• Communication: Slack
![Page 11: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/11.jpg)
11
CI Infrastructure
• ~300 jenkins jobs• 20 git repositories• On the fly jenkins slaves• Jenkins and Slack integration• Mesos cluster (Marathon,
marathon-lb, mesos-dns, Calico, chronos)
![Page 12: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/12.jpg)
Jenkins Jobs Management
![Page 13: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/13.jpg)
13
Jenkins Job Builder
Jenkins job builder to the rescue!
![Page 14: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/14.jpg)
14
Jenkins Job Builder• Developed by OpenStack folks• Configuration as code (yaml format)• Easy to review changes• Configuration de-duplication• Include shell/groovy/python… scripts• Test before deploying• Easier to organize (per directory, per file)• Serves as backup (easy to replicate to another jenkins)
![Page 15: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/15.jpg)
15
![Page 16: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/16.jpg)
16
![Page 17: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/17.jpg)
17
Templates• For nearly identical jobs better to
use templates
![Page 18: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/18.jpg)
18
Templates
![Page 19: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/19.jpg)
19
Jobs Update
![Page 20: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/20.jpg)
20
![Page 21: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/21.jpg)
21
Jenkins Jobs Types• Gating – listens for patch-set-created events• Build – for building purposes (gradle, docker etc)• Listeners – listens for change-merged events on gerrit (orchestrators for the
pipelines)
![Page 22: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/22.jpg)
22
Gating Jobs• For each patch we run a gating job• Each git project has its own gating job• Build + test + post results to gerrit
![Page 23: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/23.jpg)
23
Gating Jobs
Developer sends a patch Run build and tests(gating)
Post results to gerritMerge ?
Start build pipeline(listener)
![Page 24: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/24.jpg)
24
Gerrit• web-based code review tool built on top of the git
![Page 25: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/25.jpg)
25
Jenkins Failure
![Page 26: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/26.jpg)
26
Sonar Failure
![Page 27: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/27.jpg)
27
Gerrit Failure
Gerrit hooks• Executed on the server side• Execute per event type• Various checks: commit message style, trailing white
spaces, etc.• Integrations with external systems: bugzilla, jira, etc.
![Page 28: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/28.jpg)
CONFIDENTIAL 28
![Page 29: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/29.jpg)
Dynamic Pipelines
![Page 30: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/30.jpg)
30
Listener Jobs• Executed on patch-merged event• Orchestrating the build and delivery pipeline dynamically• Orchestration done via the BuildFlow plugin (groovy)• All listeners run the same code base• On failure, user is notified on slack channel
![Page 31: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/31.jpg)
31
![Page 32: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/32.jpg)
32
Dynamic Flows
CONFIDENTIAL 32
Listener - 1
Listener - 2
Listener - n
war
Jar
doc
docker
Test2
Mesos2
cont Mesos1
Listeners Build Deploy
…
Test
Test1
RPublish
Upload
LPublish
Bintray
Repo
![Page 33: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/33.jpg)
33
Parallel Deployments
Automation R&D Staging Production
![Page 34: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/34.jpg)
34
![Page 35: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/35.jpg)
35
![Page 36: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/36.jpg)
CONFIDENTIAL 36
![Page 37: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/37.jpg)
37
Upgrading a Stateful platform
Goals:• Minimal service interruptions• Support schema changes
Challenges:• Symmetrical cluster: Can’t refactor / add API paths• State & Business Logic in the same tier: can’t separate schema upgrade from
BL changes
![Page 38: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/38.jpg)
38
Upgrading a Stateful platform
Design:• Work in cycles, get meaningful metrics per cycle• Each cycle migrates and transforms state• Use a Threshold to determine progress and cutoff point• Smartly queue external traffic• Reroute traffic to new cluster
![Page 39: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/39.jpg)
39
Start Migration
Node
Node
Node
NodeNode
Node
Blue NodeGroup
Green NodeGroup
Create Green ClusterDiscover StateMigrate & TransformCycleCheck Threshold
Pull State
{ “documents”:”15M”, “duration”:”25S”}
{ “documents”:”15M”, “duration”:”25S”}
Migrate & TransformCycle
Pull State
{ “documents”:”6M”, “duration”:”5S”}
{ “documents”:”6M”, “duration”:”5S”}
Check ThresholdMigrate & TransformCycle
Pull State
{ “documents”:”90K”, “duration”:”0.5S”}
Check Threshold
{ “documents”:”90K”, “duration”:”0.5S”}
External Clients
Stop TrafficFinal CycleReroute Traffic
Pull State
{ “documents”:”10K”, “duration”:”0.2S”}
![Page 40: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/40.jpg)
40
Xenon – Distributed Control Plane
• A design pattern and runtime for scalable orchestration and management logic• A runtime powering tiny REST services• IO Pipeline integrates key building blocks within each service operation• Production ready code with continuous integration tests, design documents
https://github.com/vmware/xenon
![Page 41: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/41.jpg)
The Popular WayStand up N nodes for each of:
• Orchestration code & container (Spring Boot)• Your HA persistency layer (Cassandra, Mongo)• Your translation layer (ORM)• Your arbitration/leader election (ZK, etcd, consul)• Your UI server (node.js, tomcat, apache)• Your cache layer (Redis, memcached)• Your message bus, event broker
![Page 42: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/42.jpg)
The Xenon WayStand up N nodes running Xenon services:• Orchestration as stateless or stateful REST endpoints• Persist, replicate state independently• Manage concurrency with a single JVM and one thread per core across ALL
services• Provide per operation owner selection (leader)• Pub / Sub• Stats• UI• Tracing
![Page 43: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/43.jpg)
43
Links• Jenkins Jobs Builder - http://docs.openstack.org/infra/jenkins-job-builder• Xenon - https://github.com/vmware/xenon
![Page 44: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/44.jpg)
44
Thank you!
Q&A
![Page 45: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/45.jpg)
Decentralized Model• Scalable to lots of nodes
– SWIM node discovery and maintenance– Replication with Eventual OR Strong Consistency (choose!)
• Every node in a node group has the same core services– Operational simplicity
![Page 46: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/46.jpg)
Indexing/Queries• Multi version, fully indexed, replicated document store
– Lucene!
• Query services with rich document query support modeled as tasks– Real time or historical
• Collections are just queries
![Page 47: Continuous Deployment into the Unknown with Artifactory, Bintray, Docker and Mesos](https://reader035.vdocuments.mx/reader035/viewer/2022062522/587b423c1a28ab9c0e8b5ed3/html5/thumbnails/47.jpg)
Programming Model• Isolated, asynchronous components listening on URIs
• Each service instance represents a “living” document– All side effects happen through REST actions on document– Replication, consensus, notifications all leveraging symmetric model
• Stateless handlers are offered latest state and request body
• Developer declares requirements through Service options – Replication with Strong (Eager) or Eventual consistency– Scale out (Owner selection)– Instrumentation– Persistence (with deep indexing)– And more …