large scale continuous integration. made by: erik romson zenior
TRANSCRIPT
Large Scale Continuous Integration
Made by:Erik RomsonZenior
Presented by:Johannes BrodwallLead Software ArchitectBBS
(1)
“Wait a minute…”
Architect?
Reuse and cooperation
«delsystem»Delsystem1
«delsystem»Delsystem2
«delsystem»Delsystem3
«delsystem»Delsystem4
«delsystem»Delsystem5
Release strategy (one code line)
Development
When the code is stable, a branch is made
Trunk
Test Production
Development Test Production
Rel. 1.0.0 Rel. 1.0.1 Rel. 1.0.2 Rel. 1.0.3Rel. 1.0.4
Rel. 2.0.1
Ends when the new version goes into production
Principle: Everyone should be on one build server
(2)
Consequences
Size
0
50
100
150
200
250
300
350
400
450
500
Novem
ber
Desem
ber
Janu
ar
Febr
uar
Mar
sApr
il
Module count
KLOC (estimate)
Developers(estimate)
Build time
0
5
10
15
20
25
30
35
Novem
ber
Desem
ber
Janu
ar
Febr
uar
Mar
sApr
il
MinimumTypiskMaksimum
Error notification is delayed
TrunkCheckinCheckin CheckinCheckinCheckinCheckinCheckinCheckin
Skyldig
CheckinCheckinCheckinCheckin
Ändringar inkl. fel
Brutet byggBad checkin Buildstarts
Build breaks
Many developers
CruiseControlContinuous
BuildCVS
checks incode
all changes
checks incode
checks incode
checks incode
checks incode
Whodoneit?10 suspects
TrunkCheckinCheckin CheckinCheckinCheckinCheckinCheckinCheckin
Skyldig
CheckinCheckinCheckinCheckin
Ändringar inkl. fel
Brutet byggBad checkin Buildstarts
Build breaks
(3)
How does it look?
(disclaimer)
20:04 4,09m
20:15 6,15m
20:24 7,29m
20:35 8,37m
21:28 11,36m
21:53 10,41m
22:12 12,05m
(4)
How does it work?
CruiseControlContinuous
Build
CruiseControlIntegration
Build
CruiseControlDaily Build
CVS
checks incode
all changes
Uses compileand tested code
Triggers build
Installscompiled andtested code
Repository
Update tested and compiled code from repository
daily
Test ServerDeploys
applicationand run tests
foo depends onbar-1.1
Maven
Continuous build: Quickly after checkin
(should’ve caught all mistakes)
Integration build: In due time
(creates (and deploys some) EARs etc)
Daily build: Nightly
(generates project-sites)
CruiseControlContinuous
Build
CruiseControlIntegration
Build
CruiseControlDaily Build
CVS
checks incode
all changes
Uses compileand tested code
Triggers build
Installscompiled andtested code
Repository
Update tested and compiled code from repository
daily
Test ServerDeploys
applicationand run tests
Technology: Maven + Cruisecontrol + massive amounts of custom code
Implementation:Cruisecontrol triggers a custom maven plugin which uses beanshell to access a custom Java API that executes Maven goals for modules
(I think)
(5)
Lessons learned
What is hard?
Somebody has to care about the build time!
Culture for following up when the build breaks
This one is mine. I forgot to check in a file.
But sometimes, things go wrong…
Anyone looking into this one?
Could xxx please look here ASAP!
He’s gone home I will check it out.
Notification mails must be easy to interpret
Many dependencies
Buildserver
CVS Server
Mailserver
Databas
Artifactrepository
Refactoring central modules is expensive
(but doable, even in a system with several 100 KLOC)
Don’t leave the build red while you fix your problems
CVS update frequently!
Protect yourself with unit tests
(6)
The road goes ever on
Domain Driven Design build server
Module
Dependency
BuildAttempt
ChangeSet
BuildStep
*parent
*dependent
1*
*
(cruisecontrol = cron on steroids)
Smaller units of reporting
Concurrent build
Build farm
Buildconfiguration
BuildMaster
Build Worker
Build Worker
Build WorkerSource repository
Artifact repository
1. calculatechangeset
2. takework unit
3. get latest
4. publish artifact
Better visibility
