on software release engineering (bram adams)

OnSoftware Release

Engineering

Bram Adams

MC IS

Latest version: http://mcis.polymtl.ca/~bram/publications/2012/icse_tb.pdf

MC IS

(Lab on Maintenance, Construction and Intelligence

of Software)

On average we deploy new code fifty times

a day.

MC IS

Continuous Delivery

http://goo.gl/qPT6

CVS

continuousintegration

9 min.

15k tests

test

staging/production

6

Continuous Delivery

http://goo.gl/qPT6

CVS

continuousintegration

9 min.

15k tests

6 min.

test

staging/production

6

Work fast and don’t be afraid to break

things.

http://goo.gl/UlCW

James Whittaker

Build a little and then test it.

Build some more and test some more.

Even Desktop Apps Release More Frequently

2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

8

16

24

32

40

#releases

?

9 http

://en

.wik

iped

ia.o

rg/w

iki/H

isto

ry_o

f_Fi

refo

x

http://hacks.mozilla.org/w

p-content/uploads/2012/05/firefox-releases.jpg

Firefox' New Release Schedule (1)

http://hacks.mozilla.org/w

p-content/uploads/2012/05/firefox-releases.jpg

Firefox' New Release Schedule (2)

http

://ha

cks.

moz

illa.

org/

wp-

cont

ent/

uplo

ads/

2012

/05/

rapi

d-re

leas

e.jp

g Release Pipeline

Firefox Release Howto (1)

85+ repositories

configurationsanity check

tag everything

build

signingi18n

generate incremental updates for

each supported old version

Firefox Release Howto (2)

... yet Software Systems keep on Growing!

http://tims-ideas.blogspot.com

/2011/05/live-blog-from-icse-view

-of-icses.html

>5k developers

>2k projects

>50k builds/day

>50M tests/day

20 code

changes/min.compilation in the cloud

build cache with >50TB memory

<150k tests/commit

15

Release Engineering

deployment

in-house/3rd partydevelopment

http://behrns.files.wordpress.com/2008/03/ikea-car.jpg

16

integrate

buildtestreduce cycle time!

Out of Scope (but Highly Related)

17

release

management

app stores

packaging technologies

software process

impact on

project

management

variability management

software product lines

...

release planning

Release Engineering

deployment



18

integrate


19

The Build Process

?

The Build Process


20

Step 1 - Configuration

21

Tools

Autotools

Features

Step 2 - Construction

22

.cPrescriptions

.o

Dependencies

=

.exe

.o.o.o

Make

23

+

Build SystemSource Code

Build Systemsare Complex

Our record so far is a project we inherited with an Ant script weighing in at 10,000 lines of XML. Needless to say, this project required an entire team devoted to keeping the build working—a complete waste of resources.[Jez Humble & David Farley]

KDE 4 is leaving the aging "autotool" build chain behind. Some developers, not only in KDE, like to nickname the autotools as "auto-hell" because of its difficulty to comprehend architecture.[http://lwn.net/Articles/188693/] 25

>5 MLOC & ~20 Years of History

"Design Recovery and Maintenance of Build Systems" (Adams et al.)

26"The Evolution of the Linux Build System" (Adams et al.)

http://mcis.polymtl.ca/~bram/publications/2007/icsm.pdf

http://mcis.polymtl.ca/~bram/publications/2007/icsm.pdf

http://eceasst.cs.tu-berlin.de/index.php/eceasst/article/download/115/119


quake3.exe

network

d(a)emon

client UI

client renderer

server

game logic

Quake 3

27

original game

expansion

pack

Quake 3 conditional compilation

28

"The Evolution of the Linux Build System" (Adams et al.)

29

Linux 2.6.16.18

0.01




29

Linux 2.6.16.18

0.01 0.11 1.0

1.2 2.0 2.2

2.4 2.6.0 2.6.21.5




29

Linux 2.6.16.18

0.01 0.11 1.0

1.2 2.0 2.2

2.4 2.6.0 2.6.21.5

Build Systems Grow in Complexity



Build Systems Require 12%of a Developer’s

Time (on average)

30

Build maintenanceslows down development!

Software in the DOE: The Hidden Overhead of the “Build” (Kumfert et al.)



>36 MLOC & ~120 Years of History

31

An Empirical Study of Build Maintenance Effort (McIntosh et al.)

http://sail.cs.queensu.ca/publications/pubs/icse2011-mcintosh.pdf


Build Files Change Relatively More than Source Code Files

32

ArgoUML Hibernate Eclipse GCC Git Linux Mozilla PLPlot PostgreSQL0

0.51

1.52

2.53

3.54

4.55

#changed source lines/source file#changed build lines/build file

An Empirical Study of Build Maintenance Effort (McIntosh et al.)



The Build System Requires Significant Maintenance

33

4%

16%

27%

8%

20%

44%

Low coupling thanks to higher-level build

abstractionSource ⇒ BuildTest ⇒ Build

12%

34

Our Build Systems need HELP

35

A B C

D

36

A B CD

36

A B CD

static build

dependencies

all files/products stored in cloud

transparent

access via FUSE

SYMake (Tamrawi et al.)

Build System Quality

Build System Quality

Figure 2. Clipper prototype

abandoned code and the build targets that compile it can slipunder the radar. Because Google’s codebase is monolithic,these dead targets are visible to everyone. That can causepain for anyone attempting a global refactoring or even justlooking for existing functionality to reuse. Depending onexisting code helps projects move faster, but only if anengineer does not need to repair existing build breakagesto get the unmaintained code running.

This debt adds interest cost to every refactoring thataffects the dead code, as engineers waste time modifyingunused, and perhaps unusable, libraries. Broken targets addto the burden of testing a global change. There is alsoa cognitive cost to understanding a codebase where someportions are effectively unusable.

Repaying this debt involves first categorizing brokentargets as either transient breakages or long-term zombies.The results of every build and test run at Google are stored,indexed, and queryable. We use this query capability todetermine the last time that each target was successfullybuilt, as well as the last time it was attempted to be built.

Our automation updates this data nightly, and generates a listof the day’s zombie targets that existed at head in the sourcecontrol system at that time. During frequent build systemupgrades, the build team attempts to build every target inevery BUILD file in order to validate the new features,so these data are even available for completely abandonedprojects. After internal discussions, if all build attempts havefailed for at least 90 days, a target can be officially declared‘dead.’ Until then, breakages are assumed to be transient andleft to their owners, if any, to handle.

After 90 days, we can ‘terminate’ a zombie target bydeleting its definition from the BUILD file, along with anysource files that are only reachable through that target. Allcode changes, including these deletions, must be reviewedby the project’s owner of record before submission, provid-ing a final check that prevents the automation from runningamok and wiping out large swaths of Google’s live code.

V. VISIBILITY DEBT

To eliminate unwanted project coupling, where oneproject depends on the internal implementation details of

Clipper (Morgenthaler et al.)

Tu et al., Nadi et al., Berger et al., Tartler et al., ...

Release Engineering

deployment



38

integrate


The Integration Process

39

1. The Paradox of Early Integration

40


40

release branch


40

featurebranches


40

Integrations Interrupt Development once in every 24.4 Commits

41

Cohesive and Isolated Development with Branches (Barr et al.)

changed files: A, B, C, D and E

changed files: A, E and F

distractionon merge

|{A, B, C, D, E}| ∩ |{A, E, F}|

|{A, E, F}|= 66.6%

assumption: Pr[real integration conflict] = 90%

in DVC

=



... and when they Do, Integrations cause Trouble

42

Proactive Detection of Collaboration Conflicts (Brun et al.)

on average 16% of all merges had

textual conflicts (manual resolution

needed)

on average 5% of all merges had

build issues

on average 11.7% of all

merges had test issues

Branching Degrades Quality of Components

43

The Effect of Branching Strategies on Software Quality (Shihab et al.)

file branched/merged

branch family



Speculative Merge Prevention

44

Proactive Detection of Collaboration Conflicts (Brun et al.)

B T

SAME AHEAD BEHIND TEXTUAL8 BUILD8 TEST8 TESTXFigure 8: Crystal associates an icon with each of the seven relation-ships. The color of each relationship icon represents the severity ofthe relationship (Section 3.2): relationships that require no merg-ing are green, that can be merged automatically are yellow, and thatrequire manual merging are red.

solid, which combines the When and Capable guidance. If Georgecannot affect the relationship, the icon is hollow. For example, con-sider John, who has made some changes in “Let it be” but has notyet pushed them to the master; George is BEHIND John, but theicon is hollow because George cannot affect this relationship untilJohn pushes. Similarly, George’s relationship with Ringo is a hol-low TESTX because (1) George has the SAME relationship with themaster, (2) Ringo had not pulled the latest changes from the mas-ter, and (3) Ringo has made some other changes, which he has notpushed but which can merge without human intervention. If therelationship is of the might variety — George might or might nothave to perform an operation to affect the relationship — the icon issolid but slightly unsaturated (see the relationship with Bob in the“Handle with care” project).

These features allow George to quickly scan the Crystal windowand identify the most urgent issues, the solid red icons, followedby other, less severe icons. George can also easily identify quicklywhether there is something he can do now to improve his relation-ships (in the example, George can perform actions to improve hisrelationships in the “Handle with care” project, but not in “Let itbe”), and whether there are unexpected conflicts George may wishto communicate with others about.

The most urgent relationship is displayed by Crystal as its systemtray icon, which allows a developer to know at all times whetherthere is any action that requires attention without even having theCrystal window open.

Crystal also provides other guidance that is hidden unless a de-veloper wants to see it. Holding the mouse pointer over an icondisplays the action George can perform and the Committer, Conse-quences, and Ease guidance, when applicable. For example, whenGeorge holds the mouse over Jeff’s TEXTUAL8 icon, it tells himthat he can perform a pull and a resolve (hg fetch, in Mercurial),that performing this action will resolve George’s TEXTUAL8 withJeff, and that Tom and George committed the conflicting change-sets.

Even though George asked for information about the relationshipwith Jeff, Crystal was able to correctly point George to Tom asthe developer who was responsible for the conflicting changesets(which Jeff had pulled into his repository). In other situations, itis possible that George performing a pull and resolve operationswith his parent would not resolve George’s TEXTUAL8 with Jeff(e.g., if Jeff and Tom had both created conflicting changesets butonly Tom had pushed his changesets to the master). This is whythe consequences guidance is important. As a final note, becauseno one else has merged these changesets, George must resolve thisconflict and there is no Ease guidance for Crystal to display.

6.2 Initial experienceCrystal consists of 5,200 NCSL of Java and has been tested on

Windows, Mac OS X, and several Linux distributions. The de-veloper using Crystal must have read access to the collaborators’repositories; the Crystal manual (available at http://crystalvc.googlecode.com) describes several simple ways to accomplish this.

We deployed the beta-test version of the tool to a small number

of developers and have been using it ourselves, and refining it, sinceearly July 2010. One co-author uses Crystal to monitor 49 clonesof 10 projects belonging to eight actively working collaborators.

Designing and deploying Crystal, along with frequent feedbackfrom the handful of users, has helped us to better understand theissues and to improve the tool’s design. Crystal user feedbackenhanced our understanding of the need for guidance as well aswhich information is most pertinent to make available to the devel-oper. For example, showing hollow and solid icons arose from auser’s need to differentiate between relationships he or she couldand could not affect. The feedback drove us to systematically ex-plore the complete space, as described in Section 5.

Here is one example piece of feedback from an external user, viaprivate communication:

“Keeping a group of developers informed about the state ofa code repository is a problem I have tried solving myself.My solution was an IRC bot that announced commits to anIRC channel where all of the developers on the project idled.This approach has many obvious problems. [. . . ] The Crys-tal tool does not suffer from these problems. Crystal handlesseveral projects and users effortlessly and presents the nec-essary information in a simple and understandable way, butit is only a start at filling this important void the in the worldof version control.”

Prior to developing Crystal, we surveyed 50 DVCS users abouttheir collaborative development habits. Their use of highly hetero-geneous operating systems, IDEs, VCSes, and languages informedCrystal’s design. Even among this small group, there were vast dif-ferences in committing, pushing, and pulling styles, which furtherencouraged our research. We anticipate that future user studies willidentify additional strengths and weaknesses that will allow us tofurther improve Crystal.

7. Related WorkThis section places our research in the context of related work

in evaluating the costs of conflicts, collaborative awareness, miningsoftware repositories, and continuous development.

7.1 The cost of conflictsEfficient coordination is important for effective software devel-

opment. The number of defects rises as the amount of parallelwork increases [23], but developers can more effectively managerisks to the consistency of their systems if they are aware of theconsequences of their commits on other developers [11]. Develop-ers eschew parallel work to avoid having to resolve conflicts whencommitting changes [13], or rush their work into the trunk to avoidbeing the developer who would have to resolve conflicts [8].

Several observational and laboratory experiments empirically as-sess the benefits of collaborative awareness for configuration man-agement [3, 10, 30]. Augmenting these results, we performed aretrospective analysis on real projects to estimate the potential ben-efit. Our analysis is consistent with their studies in confirming thepotential for better coordination of individual and team repositories.

Sarma provides a comprehensive classification of collaborativetools for software development [27]; in this classification, Crys-tal could be considered a seamless tool as it provides continuousawareness about development state and guidance about the conse-quences of potential future actions.

7.2 Collaborative awarenessThe research most similar in intent to ours studies collaborative

awareness — increasing awareness of the activities among team

Conflict Resolution Recommendation

45

A Cost-Benefit Approach to Recommending Conflict Resolution for Parallel Software Development (Niu et al.)

(a) Using cost-benefit ratio to rank the conflicting procedures

(b) Showing details for further investigation of a particular procedure

Figure 4. Screenshots of the prototype recommendation tool.

modules or libraries. The prototype tool currently comparestwo parallel versions and relies on Semantic Diff [5] todetect semantic inconsistencies within the chosen scope.

As shown in Fig. 4a, the conflicting procedures areinitially displayed alphabetically. The recommender takesthe source code and the parallel change information asinput, computes the cost and benefit of conflict resolutionas described earlier in this section, and outputs a ranked listas a reference point for the maintainer to resolve conflicts.In case of a tie (e.g., node packi and node unpacki inFig. 4a), the procedures are listed in a random order. Inaddition to the default benefit

cost

ratio ranking, the maintainercan (re-)order the procedures by other attributes (columns)like name, cost, and benefit (cf. Fig. 4a).

To support further investigation of a particular procedure,the tool shows the detailed explanation in a separate window(cf. Fig. 4b). In this view, the dependence pairs inconsis-tent between the parallel versions are listed. These pairscontribute directly to the cost calculation in our approach.Meanwhile, the benefit of resolving the chosen procedure isexplained by providing the change propagation informationalong with the global variable(s) behind each intercon-nection. The explanation viewer integrates both semantic(i.e., cost estimation) and structural (i.e., benefit estimation)information to offer insights into the recommendations.

III. A PRELIMINARY EVALUATION

We conducted an initial evaluation of our approach bycollaborating with the computational scientists and soft-

Table ICHARACTERISTICS OF THE PARALLEL VERSIONS

HDTn.1 HDTn.2

# of procedures 928 922Mean LOC? per procedure 78.8 82.0Mean SLOC† per procedure 68.6 71.6Mean |DP| per procedure 69.0 72.4# of conflicting procedures 144nn? LOC: lines of code.† SLOC: source LOC (neither comment nor empty line is considered).

Table IIIMPACT SET SIZE OF THE CONFLICTING PROCEDURES

Alphabetic Impact Impact Our recom-order set size set size mended orderp a 2 12 p dp b 8 8 p bp c 5 8 p ep d 12 10 p fp e 8 5 p cp f 10 2 p a

ware engineers at the U.S. Army Engineer Research &Development Center (ERDC). The objective is to assess thefeasibility of our approach, and more importantly, to identifyareas for improvement. In addition, we would like to explorehow software conflicts are handled in practice.

The subject system of our study is a large-scale scientificapplication written in C. In order to honor confidentialityagreements, we use the pseudonym “HDT” to refer to thesystem. Table I lists some basic information of the twoparallel versions in our study. These HDT variants are verysimilar in terms of the statistics reported in Table I. It isinteresting to note that the |DP| : SLOC ratio per procedurein HDT is roughly 1 : 1, whereas a 4 : 1 ratio was reported(about 400 dependence pairs for a 100-SLOC procedure)when an industrial software system written in C from thereal-time domain was analyzed [5]. The relatively low ratiosuggests that fewer semantic dependencies exist in HDT thanthe system studied in [5]. This could be due to the modulardesign and regular refactoring of HDT, allowing processlibraries (e.g., sediment transport, multiple elemental cycles,etc.) to be accessed as required for a specific application.

HDT is a legacy software system that has been evolved forseveral decades. Parallel changes are common and core HDTdevelopment activities. Currently, a single maintainer takesprimary responsibility for merging HDT’s multiple versionsand variants. Although automated tools such as DiffMerge(http://www.sourcegear.com/diffmerge) are employed, the chal-lenge of conflict resolution remains. Based on an informalinterview with the HDT maintainer, when 5 to 10 inconsis-tent procedures are identified, deciding which one to resolvefirst can be challenging. In some situations, the maintainerresolves conflicts based on the order in which they show upin DiffMerge, e.g., by following an alphabetic order. Part ofthe problem is that there exists insufficient documentationof change intent. Therefore, the code base typically serves

23

how hard to resolve conflict?

how much benefit does resolving conflict yield?

ranking of methods to resolve conflicts with

method of interest

2. The Nightmare of 3rd Party Integration

46 vendor branch

Integrators are Gatekeepers

end users

47

Integrators are Gatekeepers

end users

maintainers

47

>180k Packages & ~28 Years of History

48

A B C D E F

%pa

ckag

e ve

rsio

ns0

1020

3040

5060

New Package

Upstream Sync

Dependency Management

Build Change

Product-wideConcern

Local Patch

49

Software Evolves

Mozilla Delivers New Version of Firefox – First Web Browser to

Support Do Not Track on Multiple Platforms!

[Mozilla Blog]

Necessary?

Buggy?

Secure?Upstream Sync

50

Risk Analysis & Cherry-picking

Upstream Sync

UNIX diff

maintainer

51

A B C D E F

%pa

ckag

e ve

rsio

ns0

1020

3040

5060

New Package

Upstream Sync


Build Change

Product-wideConcern

Local Patch

52

Library Transitions Ripple through the whole System Dependency

ManagementEmpir Software Eng (2009) 14:262–285 279

libz1 zlib1g

gconv–modules

libc6

debianutils

libstdc++2.10

libglib1.2 ldso

libgtk1.2

xlib6g

libjpeg62

libpng2

mozilla

libnspr4

xcontrib xfree86–common

Fig. 7 Most popular instance of the Inter-Dependency Graph for mozilla in Debian 2.2. Each of thetwo abstract dependencies have only one child

In Debian 2.0 the packages with more dependencies had 19 (python-gdk-imlib,boot-floppies and libhdf4-altdev). In Debian 4.0 the package with the largest numberof dependencies is kde, with 561, followed by gnome, with 486. kde and gnome aresets of GUI applications for the Unix desktop, none of them is present in Debian 2.0.

Both kde and gnome are bundles of packages. In practical terms this means thatthey do not have any source code associated: when these packages are installed, thebundle is installed. This raises three noteworthy issues: first, from the point of viewof the user installing such bundles, these collections of packages operate as a singlesoftware product; second, it can be argued that these packages inflate the averagenumber of dependencies without adding any new source code themselves; and third,they can be considered a great demonstration of the power of component-orientedsoftware engineering, where a “new” application, the bundle, can be created frommany components without writing a single line of code.

As the number of dependencies of packages evolves, their dependency graphs arelikely to change too. For example, Fig. 6 shows the pIDG of mozilla in Debian 4.0,which can be compared to its dependency graph in Debian 2.2, depicted in Fig. 7.Mozilla required 13 packages in 2.2 (the first version of Debian to include it), and72 in 4.0. This growth is expected as applications evolve and grow to satisfy newerrequirements.

Table 7 Evolution of thenumber of all dependencies forsome selected binary packages,for the studied Debian releases

2.0 2.1 2.2 3.0 3.1 4.0

Apache1.3 6 7 11 45 57 64Mozilla N/A 7 13 21 75 72PostgreSQL 9 16 9 23 54 42

Macro-level software evolution: a case study of a large software compilation (Gonzalez-Barahona et al.)

2-way Impact AnalysisDependency Management

Empir Software Eng (2009) 14:262–285 279

libz1 zlib1g

gconv–modules

libc6

debianutils

libstdc++2.10

libglib1.2 ldso

libgtk1.2

xlib6g

libjpeg62

libpng2

mozilla

libnspr4

xcontrib xfree86–common

Fig. 7 Most popular instance of the Inter-Dependency Graph for mozilla in Debian 2.2. Each of thetwo abstract dependencies have only one child

In Debian 2.0 the packages with more dependencies had 19 (python-gdk-imlib,boot-floppies and libhdf4-altdev). In Debian 4.0 the package with the largest numberof dependencies is kde, with 561, followed by gnome, with 486. kde and gnome aresets of GUI applications for the Unix desktop, none of them is present in Debian 2.0.

Both kde and gnome are bundles of packages. In practical terms this means thatthey do not have any source code associated: when these packages are installed, thebundle is installed. This raises three noteworthy issues: first, from the point of viewof the user installing such bundles, these collections of packages operate as a singlesoftware product; second, it can be argued that these packages inflate the averagenumber of dependencies without adding any new source code themselves; and third,they can be considered a great demonstration of the power of component-orientedsoftware engineering, where a “new” application, the bundle, can be created frommany components without writing a single line of code.

As the number of dependencies of packages evolves, their dependency graphs arelikely to change too. For example, Fig. 6 shows the pIDG of mozilla in Debian 4.0,which can be compared to its dependency graph in Debian 2.2, depicted in Fig. 7.Mozilla required 13 packages in 2.2 (the first version of Debian to include it), and72 in 4.0. This growth is expected as applications evolve and grow to satisfy newerrequirements.

Table 7 Evolution of thenumber of all dependencies forsome selected binary packages,for the studied Debian releases

2.0 2.1 2.2 3.0 3.1 4.0

Apache1.3 6 7 11 45 57 64Mozilla N/A 7 13 21 75 72PostgreSQL 9 16 9 23 54 42

Did I break someone's package?

Did someone break my package?

54

James Whittaker

If an engineer comes up with a better way of doing

something, he is tasked with refactoring all

existing libraries and assisting dependent projects to migrate

to the new libraries

A B C D E F

%pa

ckag

e ve

rsio

ns0

1020

3040

5060

New Package

Upstream Sync


Build Change

Product-wideConcern

Local Patch

56

upstream

maintainer

Upstream Sync

Local Patch

Upstream Sync

Local Patch

Upstream Sync

patchaccepted

57

Package : opensslVulnerability : predictable random number generatorProblem type : remoteDebian-specific: yesCVE Id(s) : CVE-2008-0166Date : 2008-05-13

Luciano Bello discovered that the random number generator in Debian's openssl package is predictable. This is caused by an incorrect Debian-specific change to the openssl package (CVE-2008-0166). As a result, cryptographic key material may be guessable.

It is strongly recommended that all cryptographic key material which has been generated by OpenSSL versions starting with 0.9.8c-1 on Debian systems is recreated from scratch. Furthermore, all DSA keys ever used on affected Debian systems for signing or authentication purposes should be considered compromised; the Digital Signature Algorithm relies on a secret random value used during signature generation.

The first vulnerable version, 0.9.8c-1, was uploaded to the unstable distribution on 2006-09-17, and has since propagated to the testing and current stable (etch) distributions. The old stable distribution (sarge) is not affected.

>=44 distributions ;-)

Local Patch

>=1.5 years 8-(

58

Release Engineering

deployment



59

integrate


Incremental Testing

Different Test Phases

integration + build

release

(selection of) unit tests

all unit tests

acceptance/integration tests

user acceptance tests

test phase time to run (hours)

capacity tests

smoke tests

Seamless Upgrades (1): Blue/Green Deployment

routerusers DiskDB

DiskDB

old version of system


routerusers DiskDB

DiskDB


new version of system


routerusers DiskDB

DiskDB

rollback!!



Seamless Upgrades (2): Canary Deployment

router

users

DiskDB

DiskDB

DiskDB





router

users

DiskDB

DiskDB

DiskDB





Testing Challenge 1: Parallelization

Testing Challenge 2: Incrementalizing Tests

DiskDB

Testing Challenge 3: Optimizing Test Speed

Testing Challenge 4: Data Migration

Testing Challenge 5: Process Integration

Open Challenges for Release Engineering

69

=? Certificate

of High Quality

rapid release

Do Faster Releases Improve Software Quality? - An Empirical Case Study of Mozilla Firefox (Khomh et al.)

http://post.queensu.ca/~khomhf/docs/Khomh-MSR-2012.pdf




1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 6.0

7.0 8.0

9.0

Traditional Release Cycle Rapid Release Cycle

Certificateof

High Quality

1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 6.0

7.0 8.0

9.0


release&cycle&length

vs.






13 20

1.6 1.7

0

1

2

3

4

5

Traditional Release (TR) Rapid Release (RR)

Num

ber o

f Pos

t Rel

ease

Bug

s Per

Day

Same # Post-release Bugs

643

370

0

120

240

360

480

600

720

840

960

1080

1200


Med

ian

UpT

ime

in S

econ

ds

Crashes Pop Up Earlier

vs.bug&fixing

1.0 1.5 2.0 3.0 3.5 3.6 4.0 5.0 6.0

7.0 8.0

9.0


release&cycle&length






654 159

16

6

0

10

20

30

40

50

60

70

80

90

100


Bug

Age

in D

ays

Bugs are Fixed Faster

59%

37%

67%

43%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

TraditionalRelease (TR) -

Main

Rapid Release(RR) - Main

TraditionalRelease (TR) -

Beta

Rapid Release(RR) - Beta

% o

f Bug

s Fix

ed

Proportionally Less Bugs Fixed

What about Other Projects?

Certificateof

High Quality

How to Make Software and Build Comprehension

Pragmatic?

7940

Where are the Tools?

80

RELEASE IDE

test your build!

refactor your makefiles!

what did the other team break

now ;-)

keep poking those upstream guys until they

give in :-)

Work fast and don’t be afraid to break

things.

http://goo.gl/UlCW

http://www.hipero.com/canaryunion/index.php?p=1_2_2009-Festival

We need more canaries!


a day.

MC IS


a day.

MC IS

Release Engineering

deployment



7

integrate

buildtestkeeping cycle time down


a day.

MC IS

Release Engineering

deployment



7

integrate



21

8%

20%

44%

4%

16%

27%

Source ⇒ BuildTest ⇒ Build

12%


a day.

MC IS

Release Engineering

deployment



7

integrate



21

8%

20%

44%

4%

16%

27%

Source ⇒ BuildTest ⇒ Build

12%

Risk Analysis & Cherry-picking

Upstream Sync

diff

maintainer

http://google-engtools.blogspot.ca/

http://mcis.polymtl.ca/

MC IS

http://google-engtools.blogspot.ca








on software release engineering (bram adams)

Technology

integration6 http

releases http

build process19

release engineeringhttp

linux build system adams

intelligenceof software

software systemskeep

software process i