yet another continuous integration story
DESCRIPTION
PHP User Group Minsk #4TRANSCRIPT
![Page 1: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/1.jpg)
Yet Another Continuous Integration Story
![Page 2: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/2.jpg)
Who I Am● Anton Serdyuk
● Technical Leader at Startup Labs, Inc.
● @anton_serdyuk
● Skype: serdyuk.anton
![Page 3: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/3.jpg)
Disclaimer
![Page 4: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/4.jpg)
Disclaimer
Projects are different● Technical requirements
● Business requirements
● Length
● Budget
● Load
● Data size
● etc
![Page 5: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/5.jpg)
Disclaimer
Teams are different● Level
● Size
● Structure
● etc
![Page 6: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/6.jpg)
Disclaimer
There is no silver bullet● Worked for us != will work for you
● Adopt patterns and principles rather than the full
process
![Page 7: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/7.jpg)
Who We Are
![Page 8: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/8.jpg)
Who We Are
Project● b2b system
● Unexpected downtime is very expensive
● Bugs in production are also very expensive because of
the reputation loss
● Small amount of traffic (But expensive)
● 1.5 years
● 4-15 developers (+ layout, + qa)
● 15 Gb Database (Grows not very fast)
![Page 9: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/9.jpg)
Who We Are
Technologies● PHP 5.4
● Java 7
● PostgreSQL
● RabbitMQ
![Page 10: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/10.jpg)
Who We Are
Frameworks● Hibernate
● JUnit
● Symfony2
● Silex
● Behat
● PHPUnit
![Page 11: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/11.jpg)
Goals and Principles
![Page 12: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/12.jpg)
Goals and Principles
Agile● Small changes
● Frequent releases
● Early feedback
![Page 13: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/13.jpg)
What Is NOT Our Way
![Page 14: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/14.jpg)
What Is NOT Our Way
Git Flow
![Page 15: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/15.jpg)
What Is NOT Our Way
Feature Branches
master
feature2
feature1
![Page 16: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/16.jpg)
What Is NOT Our Way
Feature BranchesActive feature branches usage example
![Page 17: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/17.jpg)
What Is NOT Our Way
Release Branches
master
1.0.x
1.1.x
![Page 18: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/18.jpg)
What Is NOT Our Way
master/staging/develop branches
master
develop
![Page 19: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/19.jpg)
What Is NOT Our Way
Release Tags
master
v1.1.2v1.1.1
![Page 20: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/20.jpg)
What IS Our Way
![Page 21: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/21.jpg)
What IS Our Way
Continuous Integration● Push every day
● Build every commit
● Run all tests on every build
● Fix red builds quickly
![Page 22: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/22.jpg)
What IS Our Way
One Branch
master
![Page 23: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/23.jpg)
What IS Our Way
Branch by Abstraction● Commit switch-off for this feature
● Develop, integrate
● … your code in production now, but switched-off
● Develop, integrate
● Test
● Switch-on this feature in production
![Page 24: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/24.jpg)
What IS Our Way
Autotests● QAs write behat scenarios
● Developers implement
● Every feature covered by behat scenarios
● Great regression tests suite - easy refactoring, less
bugs
● + ~100% development cost
![Page 25: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/25.jpg)
What IS Our Way
Every Build Is Release Candidate● Do not push if state after commit is not suitable for
production
![Page 26: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/26.jpg)
What IS Our Way
Some Numbers● ~1000 behat scenarios (unit tests are not counted)
● ~15 minutes build duration
● 2 build servers (Intel Xeon, 32 Gb RAM, SSD)
● 5-6 build agents for tests (VMs)
● 3 build agents for build phase (VMs)
● ~15 builds per day
● 2 releases per week
![Page 27: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/27.jpg)
Build Process
![Page 28: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/28.jpg)
Build Process
Overview
build automatic tests manual tests load and prod migrate tests production
![Page 29: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/29.jpg)
Build Process
Overview
testagentstest
agentstest
agentsbuild agentgit
repository
push
bamboo
pull
build agents
build
aptrepository
artifacts
testagents
puppetmasterdevelopers
release engineer (architect/admin)
tune
test
QAserver
preprod prod
![Page 30: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/30.jpg)
Build Process
Use The Same Artifacts
testagentstest
agentstest
agentsbuild agentbuild
agents
aptrepository
artifacts
testagents
QAserver
preprod prod
![Page 31: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/31.jpg)
Build Process
Autoconfigure All Environments
testagentstest
agentstest
agentstest
agents
puppetmaster
release engineer (architect/admin)
tune
QAserver
preprod prod
![Page 32: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/32.jpg)
Build Process
Deploy by Pressing a Button
testagentstest
agentstest
agents
bamboo
testagents
QAserver
preprod prod
![Page 33: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/33.jpg)
Build Phase
![Page 34: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/34.jpg)
Build Phase
Goals● Create artifacts which will be used in all phases
● Run unit tests
![Page 35: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/35.jpg)
Build Phase
Technologies● Phing
● Maven
● deb-packages (dpkg-buildpackage)
● PHPUnit
● JUnit
![Page 36: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/36.jpg)
Build Phase
Example
![Page 37: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/37.jpg)
Automatic Test Phase
![Page 38: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/38.jpg)
Automatic Test Phase
Goals● Run functional tests
![Page 39: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/39.jpg)
Automatic Test Phase
Deploy to Completely Clean System
![Page 40: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/40.jpg)
agent 1
Automatic Test Phase
Run Tests in Parallel
behat process 1 application 1
...
agent 2
behat process 2 application 2
agent 5
behat process 5 application 5
![Page 41: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/41.jpg)
Automatic Test Phase
Run Tests in Parallelfeature 1
feature 2
feature 3
feature 4
feature 5
feature 6
feature 7
feature 8
feature 9
feature 50
feature 51
...
job 01
job 02
job 03
Job 15
agent 1
agent 2
agent 3...
![Page 42: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/42.jpg)
Automatic Test Phase
Run Tests in Parallelgithub: StartupLabs/behat-partial-runner
● bin/behat --count 15 --current 1
● bin/behat --count 15 --current 2
● bin/behat --count 15 --current 3
● bin/behat --count 15 --current 4
● bin/behat --count 15 --current 5
● …
● bin/behat --count 15 --current 15
![Page 43: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/43.jpg)
Automatic Test Phase
Write Fast Tests● Continuously speed-up your tests
● Use curl instead of Selenium
● Put initial data directly to DB
● etc
![Page 44: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/44.jpg)
Automatic Test Phase
Write Fast TestsMerge scenarios which do not modify data
Feature: Posts pagination
Background:
Given I am logged in as admin
And there is many posts
Scenario: Pagination works
Given I am at the posts list
Then I should see pagination
Scenario: Link “last” works
Given I am at the posts list
When I click at the “last” link
Then I should see last page
Feature: Posts pagination
Background:
Given I am logged in as admin
And there is many posts
Scenario: All pagination tests
# Scenario: Pagination works
Given I am at the posts list
Then I should see pagination
# Scenario: Link “last” works
When I click at the “last” link
Then I should see last page
![Page 45: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/45.jpg)
Automatic Test Phase
Write Fast TestsTranslate outline to single scenarios with TableNode
Feature: Post validation
Background:
Given I am logged in as admin
And I am on post add page
Scenario Outline: Validation works
When I enter <value> into <field>
Then I should see error <error>
Examples:
| field | value | error |
| title | | Can not be empty |
| tags | %%%% | Should be letters |
Feature: Post validation
Background:
Given I am logged in as admin
And I am on post add page
Scenario: Validation works
Then I should see such errors with such input
| field | value | error |
| title | | Can not be empty |
| tags | %%%% | Should be letters |
![Page 46: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/46.jpg)
server
Automatic Test Phase
Write Fast Tests
@parallel
Feature: Login
Background:
Given I am registered user
| username | alex |
| password | 123 |
Scenario: I can login
Given I am at the login page
When I fill form with
| username | alex |
| password | 123 |
Then I should be logged in
Parallel test execution
● Many problems with race
conditions
● Not worked for us :(
behat process 1
behat process 2application
alex = alex1
alex = alex2
![Page 47: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/47.jpg)
Manual Test Phase
![Page 48: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/48.jpg)
Manual Test Phase
Goals● Run manual tests
● Place where QA see new functionality
![Page 49: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/49.jpg)
Manual Test Phase● Deploy to completely clean system
![Page 50: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/50.jpg)
Load and Prod Migrate Tests
![Page 51: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/51.jpg)
Load and Prod Migrate Tests
Load Tests Goals● Catch race conditions
● Catch serious performance degradation when database
grows or after adding new functionality
![Page 52: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/52.jpg)
Load and Prod Migrate Tests
Prod Migrate Tests Goals● Test migrations on real data
● Test migrations time on real data
![Page 53: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/53.jpg)
Load and Prod Migrate Tests● Migrate load test env to new version
● Rollback migrate test env to production snapshot
![Page 54: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/54.jpg)
Deploy to Production
![Page 55: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/55.jpg)
Deploy to Production● nb-prod - small production copy used by other teams for
integration purposes
![Page 56: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/56.jpg)
Problems
![Page 57: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/57.jpg)
Problems
Red Builds for a Long Time● QA can not deploy new version to staging
● New commits make new tests red like a snowball
● Nobody knows who are responsible for all those failed
tests
Solutions:
● If build can not be fixed in 15 minutes - revert commit
● Autoreverts on red builds (?)
![Page 58: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/58.jpg)
Problems
Long Builds● Nobody wants to wait green build
● Nobody cares about green build after commit
● Red builds for a long time
Solutions:
● Speedup tests
● Parallel build
● Add more build servers
![Page 59: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/59.jpg)
Problems
Blinking Failures● Failures are trustless - “it is not my fault - it is blinking,
just do rebuild”
● Time consuming
Solutions:
● Eliminate them ASAP
● Do not just run “Rebuild failed jobs”
![Page 60: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/60.jpg)
Key Points, Again
![Page 61: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/61.jpg)
Key Points, Again
Autotests● There is no sense in CI without tests
● Easy refactoring
● Less bugs
![Page 62: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/62.jpg)
Key Points, Again
Fast Builds
● Fast tests
● Parallel run
● Hacks in tests
● etc
● Long builds => More changes => Red builds
![Page 63: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/63.jpg)
Key Points, Again
One branch● There are no builds from other branches
● There are no hotfixes
● Easy deploy and build pipeline
![Page 64: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/64.jpg)
Todo
![Page 65: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/65.jpg)
Todo● Rollback strategy
● Use Bamboo deployment
![Page 66: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/66.jpg)
Inspired By● Continuous Delivery: Reliable Software Releases
through Build, Test, and Deployment Automation (Jez
Humble, David Farley)
● http://timothyfitz.com/
![Page 67: Yet Another Continuous Integration Story](https://reader034.vdocuments.mx/reader034/viewer/2022052321/554bca90b4c9058f6c8b45cf/html5/thumbnails/67.jpg)
Questions?