seacon continuous delivery pipeline tools track
TRANSCRIPT
SEACON:UK 2017 Breakout Track
Exploring different approaches to Continuous Delivery Pipelines
Diversion into Inclusion & Diversity1.It should be down to all of us to champion this at every
opportunity.2.We all suffer from Unconscious Bias and even if this has just a
0.1% on the opportunities that we give others, cumulatively this stacks up. (Take a test: https://implicit.harvard.edu/implicit/selectatest.html)
3.Mirco aggression is a term for behaviour that intimidates others and make it harder for them to perform a their best in the working environment. Be vigilant and call it out. (Real examples here: http://www.microaggressions.com/)
Other links:• https://www.fastcompany.com/3044738/hit-the-ground-running/7-simple-methods-to-fight-against-your-
unconscious-biases• https://codeascraft.com/2016/10/19/being-an-effective-ally-to-women-and-non-binary-people/• http://weprinciples.org/files/attachments/
WHITE_PAPER_Nudging_the_Unconscious_Mind_Nielsen_&_Kepinski_Oct_2014.pdf• https://managingbias.fb.com/• http://www.cmu.edu/news/stories/archives/2015/june/bias-blind-spot.html
Agenda• Introduction to Continuous Delivery In Practice
• This Exercise
• XebiaLabs - XL Deploy and XL Release
• ADOP and Jenkins
• Thoughtworks GoCD
• General Q&A
The Objective: Continuous Delivery of Ideas
FEEDBACK
BuildStatic Code Unit Tests
Source CodeRepo
CI pipeline
Deploy Code
Run Test Harness
Deploy CodePPE
Perf,Security
TestDeploy
Prod
CD pipeline(rest of the way to Prod)
Automating the Workflow from Commit to Release
The Workflow needs to hold up
How do we control flow?
What versions
were tested
together?
What should
we release?
Tests configurationBuild tools configuration
Deployment software configurationMiddleware configuration
Operating System configuration Virtual infrastructure configuration
Hardware configuration
Business
Application
Business
Application
Business
Application
Business
Application
Business
Application
Compile Code AnalysisCommitter: jdoe
Story:25
Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test
Compile Code AnalysisCommitter: jdoe
Story:25
Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test
Compile Code AnalysisCommitter: jdoe
Story:25
Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test
Compile Code AnalysisCommitter: jdoe
Story:25
Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test
Compile Code AnalysisCommitter: jdoe
Story:25
Commit ID: 113 Unit Test Deploy PPE Functional Test Security and Perf Test
We should probably do this all the way down!
Commits of any Source Code
triggers a pipeline
???
One pipeline won’t work - the tools need handle complexity
BuildStatic Code Unit Tests
Deploy Code
Run Test
Harness
Deploy CodePPE
Perf,Securit
y Test
Deploy Prod
Luckily in 2017 they all do now support a lot things e.g.:
• Agents for secure remote execution on different platforms
• Application security
• Role Based Access Control
• Reporting and metrics
• Automated and manual triggers and flow control
• Commercial support
• Extensibility
But how do they differ?
THE EXCERCISE
Application DesignReference Application Stack• Frontend App• Backend App (tightly coupled to frontend – both must be released
together)• Configuration Management = Chef or Puppet• Provisioning = TerraformOrchestration & Automation Scripts = (you choose) Tools• Software configuration Management = (you choose but not of
concern)• Binary storage = (you choose but not of concern)• Testing = (you choose and but of concern)• Monitoring = (you choose but not of concern)• Automation of build = (you choose but not of concern)• IaaS platform = (you choose but not of concern)Automation of deployment = (you choose)Continuous Delivery Pipeline = (you choose)
Example PipelineFrontend
Code, Config
BackendCode, Config
Config MgmtScripts
ProvisioningScripts
Orchestration& Automation
Scripts
Build & Package
Unit TestCode Analysis
Build & Package
Unit TestCode Analysis
Unit Test &Code Analysis
Sandbox Testing(various)
Install &ConfigureNon-Prod
Runtime Testing(various)
Install &ConfigureProduction
Unit Test &Code Analysis
Sandbox Testing(various)
ProvisionNon-Prod
Runtime Testing(various)
ProvisionProduction
Unit Test &Code Analysis
Sandbox Testing(various)
UpdateJobs /Flows
MonitoringBackupHealingetc.
System TestDeploy
System TestDeploy
Install &ConfigureSystem Test
Runtime Testing(various)
Non-ProdDeploy
Runtime Testing(various)
ProductionDeploy
Runtime Testing(various)
Runtime Testing(various)
ProvisionSystem Test
Runtime Testing(various)
FrontendCode, Config
BackendCode, Config
Config MgmtScripts
ProvisioningScripts
Orchestration& Automation
Scripts
Build & Package
Unit TestCode Analysis
Build & Package
Unit TestCode Analysis
Unit Test &Code Analysis
Sandbox Testing(various)
Install &ConfigureNon-Prod
Runtime Testing(various)
Install &ConfigureProduction
Unit Test &Code Analysis
Sandbox Testing(various)
ProvisionNon-Prod
Runtime Testing(various)
ProvisionProduction
Unit Test &Code Analysis
Sandbox Testing(various)
UpdateJobs /Flows
MonitoringBackupHealingetc.
System TestDeploy
System TestDeploy
Install &ConfigureSystem Test
Runtime Testing(various)
Non-ProdDeploy
Runtime Testing(various)
ProductionDeploy
Runtime Testing(various)
Runtime Testing(various)
ProvisionSystem Test
Runtime Testing(various)
Visualisation and release Tracking1. How can “what is deployed where” be visualised and tracked (over time) and how can tested combinations of components be tracked and locked-in?
2. How can it be enforced that a consistent use of automation is used across deployments to different environments?
FrontendCode, Config
BackendCode, Config
Config MgmtScripts
ProvisioningScripts
Orchestration& Automation
Scripts
Build & Package
Unit TestCode Analysis
Build & Package
Unit TestCode Analysis
Unit Test &Code Analysis
Sandbox Testing(various)
Install &ConfigureNon-Prod
Runtime Testing(various)
Install &ConfigureProduction
Unit Test &Code Analysis
Sandbox Testing(various)
ProvisionNon-Prod
Runtime Testing(various)
ProvisionProduction
Unit Test &Code Analysis
Sandbox Testing(various)
UpdateJobs /Flows
MonitoringBackupHealingetc.
System TestDeploy
System TestDeploy
Install &ConfigureSystem Test
Runtime Testing(various)
Non-ProdDeploy
Runtime Testing(various)
ProductionDeploy
Runtime Testing(various)
Runtime Testing(various)
ProvisionSystem Test
Runtime Testing(various)
Developing and Testing Automation3. What does the activity of creating automation and orchestration entail (Coding, DSL, Drag and Drop etc.)?
4. How is automation and orchestration tested and new versions released?
5. How can people share and re-use automation packages?
FrontendCode, Config
BackendCode, Config
Config MgmtScripts
ProvisioningScripts
Orchestration& Automation
Scripts
Build & Package
Unit TestCode Analysis
Build & Package
Unit TestCode Analysis
Unit Test &Code Analysis
Sandbox Testing(various)
Install &ConfigureNon-Prod
Runtime Testing(various)
Install &ConfigureProduction
Unit Test &Code Analysis
Sandbox Testing(various)
ProvisionNon-Prod
Runtime Testing(various)
ProvisionProduction
Unit Test &Code Analysis
Sandbox Testing(various)
UpdateJobs /Flows
MonitoringBackupHealingetc.
System TestDeploy
System TestDeploy
Install &ConfigureSystem Test
Runtime Testing(various)
Non-ProdDeploy
Runtime Testing(various)
ProductionDeploy
Runtime Testing(various)
Runtime Testing(various)
ProvisionSystem Test
Runtime Testing(various)
Build & Package
Unit TestCode Analysis
System TestDeploy
Operational Concerns7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?
6. How can the platform be extended e.g. Via Plugins?
The main event!
• Ben Hählen – Xebia Labs – XL Deploy and XL Release
• Hitesh Joshi – Accenture – ADOP and Jenkins
• Marius Ciotlos – Betfair – Thoughtworks GoCD
XL Release and XL Deploy
XL Release
XL Release
XL Release
XL Release
XL Deploy
XL Deploy
XL Deploy
To facilitate comparison we would like to propose the following scenario. The objective of the exercise is to design (no need to implement) a solution for achieving continuous delivery of the following.
Reference Application Stack• App 1 = Front-End• App 2 = Back-End• Configuration Management = Chef or Puppet• Provisioning = Terraform Orchestration & Automation Scripts = XL Release Tools• Software configuration Management = Git, SVN• Binary storage = Nexus, Artifactory, etc• Testing = Sonar, Cucumber, etc• Monitoring = ELK, Splunk, etc• Automation of build = Jenkins, VSTS, TFS, etc• IaaS platform = On-premise, Azure, Kubernetes, Docker, etc Automation of deployment = XL Deploy Continuous Delivery Pipeline = XL Release
Application Design
1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?
1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?
1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?
1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?
2. How can it be enforced that a consistent use of automation is used across deployments to different environments?
3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?
3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?
4. How is automation and orchestration tested and new versions released?
5. How can people share and re-use automation packages?
6. How can the platform be extended e.g. via plugins?
7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?
Short Term Support : Every 2 months
Long Term Support : Every 6 monthsMaintenance Release : Every 2 months
THANKS
About Me
RAPID MOBILISATION
GLOBAL SHARED SERVICES TEAM
SUPPORTING 24 / 7
Software Configuration Management
Release ManagementAssessments
Environment Management
DevOps TransformationBuild & Deploy Automation Continuous Delivery
Rational and Jira
Infrastructure as code CloudPaaS Tooling
Jenkins
Jenkins with ADOP• All open source• Commercial Jenkins option though CloudBees• Accenture created ADOP so we can help
1: December 2016: https://www.cloudbees.com/jenkins/about
Stats• 150,000 Jenkins active installations, 1M users, 12.5M jobs (1)
• Jenkins plugins: 1200+• Jenkins Pipeline installations: 77,000+• ADOP: ~550 forks + ~200 stars• ADOP plugins (cartridge) > 50
On Premise
Docker Machine
Docker Compose
ADOP Platform Extension
ADOP Cartridge
AWS Cloud
AWS regionAvailability
Zone
Internet
Selenium
Platform for Reusable Automation
Cartridges
Solution ComponentsReference Application Stack• App 1 = Hybris• App 2 = SOLR• Configuration Management = Chef or Puppet• Provisioning = Terraform Orchestration & Automation Scripts = ADOP, Jenkins, Ansible
Tools• Software configuration Management = Git• Binary storage = Nexus (with Lifecycle)• Testing = Cucumber, Selenium Grid• Monitoring = Sensu and ELK• Automation of build = Ant and Maven• IaaS platform = AWS Automation of deployment = Ansible Continuous Delivery Pipeline = ADOP, Jenkins
Visualisation and release tracking1. How can “what is deployed where” be visualised and tracked over time and how can tested combinations of components be tracked and locked-in?
Components
Environments
Visualisation and release tracking2. How can it be enforced that a consistent use of automation is used across deployments to different environments?
Developing and Testing Automation3. What does the activity of creating automation and orchestration entail (coding, DSL, drag and drop etc.)?
$ tree.|-- README.md|-- jenkins| `-- jobs| |-- dsl| | |-- environment_provisioning.groovy| | `-- hybris_jobs.groovy`-- src `-- urls.txt
Developing and Testing Automation5. How can people share and re-use automation packages?
A Cartridge encapsulates a reference complete implementation for a particular technology or product, typically consisting of:• Reference application• Build/Deploy Pipeline• Environment Creator – depending on technology
Hybris Cartridge
CreateEnvironment(AWS EC2,
RDS) Base Hybris Build
(Platform + Code) Deploy Hybris
Docker image with config +
code
Developing and Testing Automation4. How is automation and orchestration tested and new versions released?
Same Continuous Delivery Pipeline for Jenkins and Ansible
Operational Concerns6. How can the platform be extended e.g. via plugins?
Operational Concerns7. How are the Continuous Delivery Pipeline tools upgraded / patched and typically how often does this happen and why?
THANKS
THANKS