agile mëtteg - june 2011

Software Dev. Practices – Continuous Integration

Agile Mëtteg – June 16th, 2011

Agile Mëtteg – Continuous Integration 2

ABOUT US

16 June 2011

PROFILE

Created in 2004Independent Software Development Company

16 June 2011 Agile Mëtteg – Continuous Integration 3

FIGURES

16 June 2011 Agile Mëtteg – Continuous Integration 4

2004 2005 2006 2007 2008 2009 20100 €

500,000 €

1,000,000 €

1,500,000 €

2,000,000 €

2,500,000 €

3,000,000 €

Turnover

16%

1%

42%

41%

Turnover Distribution

Place FinancièreActivités IndustriellesServices & VenteSecteur public & Associatif

2,5M€ en 2010

2004 2005 2006 2007 2008 2009 2010 2011-5

0

5

10

15

20

25

30

35

0 -1 0 -1 0 -2 -1 00 29 12 13 16

2227

2

8

32 3

8

6

5

Employees turnover

32 in 2011

MISSION

Design, Develop and Customize “Software as Business & Operational Enabler”

Fast & flexible solutions business value oriented

Help IT and all Business & Operational Organizations to adopt the culture of “Software as Business & Operational Enabler”

Simple & pragmatic methods making effective the collaboration of the actors of a projectEasy and powerful tools for follow-up of relevant KPI

16 June 2011 Agile Mëtteg – Continuous Integration 5

OUR SERVICES

MgtTeam

Services

SoftwareDevelopme

nt

OpsTeam

Services

Dev Team

Services

123

4

Applications enabling productivity

Bespoke application

Mobile application

Based on package

Consulting services Coaching & SupportTrainingResource delegation

16 June 2011 Agile Mëtteg – Continuous Integration 6

1

2

3

4

OUR MEANS

16 June 2011 Agile Mëtteg – Continuous Integration 7

80% > 4 years56% > 8 years

31% > 12 years

Agility

Authorized Training Center in Luxembourg

0-4 4-8 8-12 12-16 16- 0

2

4

6

8

10

5 68

64

Overall seniority

OUR MAIN CUSTOMERS

16 June 2011 Agile Mëtteg – Continuous Integration 8

Agile Mëtteg – Continuous Integration 9

SPEAKERS

16 June 2011

Jean-Pol LANDRAIN

Sylvain CHERY

Consultant DirectorCSM - CSP

Agile Mëtteg – Continuous Integration 10

ABOUT YOU

16 June 2011

Agile Mëtteg – Continuous Integration 11

PARTICIPANTS

Who are you ?

What is your role ?

What do you know about agility ?

16 June 2011

Agile Mëtteg – Continuous Integration 12

INTRODUCTION

16 June 2011

CONTINUOUS INTEGRATION

Continuous Integration in a few questions

Why do I need it ?

What is it ?

What does it require ?

How does it relate to the Agile principles ?

16 June 2011 Agile Mëtteg – Continuous Integration 13

SOFTWARE DEVELOPMENT: INDUSTRY? PROCESSES?

What is Software Development?

an industry mining, farming, construction, manufacturing,

…etc.

with clearly identified and well-defined processes, i.e. easily reproducible

Really?No failed project?A lot of…

16 June 2011 Agile Mëtteg – Continuous Integration 14

SOFTWARE DEVELOPMENT: A LOT OF FAILED PROJECTS

Gartner

75% of failure on at least one of

Cost Quality Time

16 June 2011 Agile Mëtteg – Continuous Integration 15

SOFTWARE DEVELOPMENT: A LOT OF FAILED PROJECTS

Source Result

KPMG Canada – Survey 1997

61% of failure, out of which:

• 30% over time• 50% tremendously over

budget

Standish Group (US) – Survey 2008

16,2% of success in all aspects:

• cost• time• deliverables

« Improbable success » in 68% of the cases

OASIG Study (UK) 1995 70% of failure « in some respect »

16 June 2011 Agile Mëtteg – Continuous Integration 16

« Houston, we’ve got a problem! »

16 June 2011 Agile Mëtteg – Continuous Integration 17

AGILE MANIFESTO vs SOFTWARE INDUSTRIALIZATION ?

Agile Manifesto

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan

16 June 2011 Agile Mëtteg – Continuous Integration 18

SOFTWARE CRAFTSMANSHIP vs SOFTWARE INDUSTRIALIZATION?

Manifesto for Software CraftsmanshipNot only working software, but also

well-crafted softwareNot only responding to change, but also

steadily adding valueNot only individuals and interactions, but also

a community of professionalsNot only customer collaboration, but also

productive partnerships

16 June 2011 Agile Mëtteg – Continuous Integration 19

SOFTWARE DEVELOPMENT: AN ART?

Agile Manifesto and Manifesto for Software Craftsmanship were created by veterans of the software industry

However, when summed up, one can conclude software development is more an art than an industrial process

So, let’s compare with music…16 June 2011 Agile Mëtteg – Continuous Integration 20

SOFTWARE DEVELOPMENT: AN ART?

16 June 2011 Agile Mëtteg – Continuous Integration 21

SOFTWARE DEVELOPMENT: AN ART?

16 June 2011 Agile Mëtteg – Continuous Integration 22

SOFTWARE DEVELOPMENT IS AN ART

So, Software Development is an art!

But building nearly anything is also an art

Don’t you think?

And, more importantly,

Art is not without rules and best practices

16 June 2011 Agile Mëtteg – Continuous Integration 23

RULES AND BEST PRACTICES

What’s the worst?

Not following the rules usually leads to a rather direct and abrupt failure

• project fails integration testing • project is refused by infrastructure• project fails user acceptance testing• …

16 June 2011 Agile Mëtteg – Continuous Integration 24

RULES AND BEST PRACTICES

What’s the worst?

Not following the best practices augments, sometimes dramatically, the risks of failure

• reduces overall quality• increases the time to market / time to

deliver• has typically a pervasive effect: can be

ignored or remains unknown until it becomes really critical

• increases the maintenance and ownership costs

16 June 2011 Agile Mëtteg – Continuous Integration 25

RULES AND BEST PRACTICES

What’s the worst?

Both are terrible: sources of failureBut one is harder to detect than the other

Necessity to put in place and to define the structures and infrastructures required to check the quality at every level

Because “the earlier, the best” (and less expensive)

16 June 2011 Agile Mëtteg – Continuous Integration 26

Agile Mëtteg – Continuous Integration 27

SOFTWARE QUALITY

16 June 2011

WHAT IS SOFTWARE QUALITY?

“Quality is value to some person” (Gerald Weinberg, “Quality Software Management”)

i.e. quality is inherently subjective; people experience the quality of the same software very differently

this applies mainly to the quality of a software product, as perceived from an external view; and it can also comprise the quality of its running environment(s)

but the quality of the source code can also have an impact on the efforts needed for having a software product that fulfills the requirements, the intrinsic quality

Steve McConnell defines external and internal quality characteristics (‘”Code Complete”)

16 June 2011 Agile Mëtteg – Continuous Integration 28

IMPLICIT FACTORS FOR SOFTWARE QUALITY

The software projects typically follow rather detailed requirement plans

Though a set of characteristics often goes unmentioned

These are the implied requirements that are expected of all professionally developed software

16 June 2011 Agile Mëtteg – Continuous Integration 29

IMPLICIT FACTORS FOR SOFTWARE QUALITY

Conformance to implied requirements:

UnderstandabilityCompletenessConcisenessPortabilityMaintainabilityTestabilityUsabilityReliabilityEfficiencySecurity

16 June 2011 Agile Mëtteg – Continuous Integration 30

GUARANTEEING CODE QUALITY

How can we guarantee the level of quality of software during the « coding » phases?

The same ways as in the industry

• Factory inspections: “static code analysis” in IT

• Incremental improvements: “release soon, release often” in IT

• Tests, tests, tests, tests, tests, tests, …

16 June 2011 Agile Mëtteg – Continuous Integration 31

GUARANTEEING CODE QUALITY

Focus on Unit Tests and Test Driven Development (TDD)

Unit Tests

• The first tests to write• Written by the developers before the code• Write the code afterward to make the tests succeed• Must run in isolation, without any infrastructure• Must be fast to execute

How many unit tests per method ?

• One test per degree of “cyclomatic complexity”

16 June 2011 Agile Mëtteg – Continuous Integration 32

GUARANTEEING CODE QUALITY

Cyclomatic complexity of a method?

• the number of linearly independent paths in the method

If .. then If .. then .. else If .. and .. then If .. or .. then

Do .. While While .. Do Switch

16 June 2011 Agile Mëtteg – Continuous Integration 33

GUARANTEEING CODE QUALITY

Necessity to put in place the supporting

Structures (practices):

• Unit testing• Integration testing• Performance testing• Regression testing• Acceptance testing

Infrastructures (tools):

• Source Code management• Issue tracking• Build tools• Continuous Integration server• Quality reporting tools

Build Industrialization

Build IndustrializationPlatform

Ag

ility

Ag

ility

16 June 2011 Agile Mëtteg – Continuous Integration 34

GUARANTEEING CODE QUALITY

How to put in place the supporting

Structures (practices):

• Define and prioritize the quality requirements• Refine and adapt the relevant quality criteria to the objectives• Communicate rules and best practices: sharing, training and mentoring• Verify the correct usage and level of adoption• Review the results and improve the processes

Infrastructures (tools):

• Select the tools adapted to the defined quality requirements• Set them up based on the selected criteria (define measurements)• Communicate on their usage: sharing, training and mentoring• Generate quality reports and metrics• Analyze the conformance of the results and adapt the tools

Ag

ility

Ag

ility

16 June 2011 Agile Mëtteg – Continuous Integration 35

Agile Mëtteg – Continuous Integration 36

CI RULES AND PRE-REQUISITES

16 June 2011

CI RULES AND PRE-REQUISITES (1/3)

maintain a code repository

automate the build

the build must be self-testing

regular code sharingi.e. frequent commits

16 June 2011 Agile Mëtteg – Continuous Integration 37

CI RULES AND PRE-REQUISITES (2/3)

self-contained modificationsi.e. commits don’t break the build process

the build must be fast

integration tests (for the least) in a clone of the production environment

16 June 2011 Agile Mëtteg – Continuous Integration 38

CI RULES AND PRE-REQUISITES (3/3)

latest deliverables easily available to anyone who needs them

easy access to the results of the tests

automated deployments to a live test server

continous deployment to production is the ideal achievement

16 June 2011 Agile Mëtteg – Continuous Integration 39

Agile Mëtteg – Continuous Integration 40

BUILD INDUSTRIALIZATION PLATFORM

16 June 2011

BUILD INDUSTRIALIZATION PLATFORM

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 41

SOURCE CODE MANAGEMENT

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 42

SOURCE CODE MANAGEMENT

What are SCM tools ?

Tools that allow sharing and versioning files… but really not their primary interest …

… shared folders can do that too !

Tools to track and document the changes in the code, in such a way the developers can work in a task or

issue oriented mode

16 June 2011 Agile Mëtteg – Continuous Integration 43

SCM RULES FOR CONTINUOUS INTEGRATION

Tag your releases

Use branching strategies

Release branches

Feature branches for large changes

“branch by abstraction” pattern works well with continuous integration (for non-distributed SCM’s like CVS, SVN, TFS…)

16 June 2011 Agile Mëtteg – Continuous Integration 44

SCM RULES FOR CONTINUOUS INTEGRATION

Commit “atomically”

all the files related to a single task/issue in one operation or, if really not possible, max a few operations

don’t mix changes from different tasks/issues in the same commit

add a meaningful, standardized comment that (for example) includes:

• the task/issue identifier (first line)• status (first line)

« completed » or « in progress »

• short description from the issue tracking system (first line)• eventually a dedicated URL from the issue tracking system (second

line)• a meaningful description of what’s been done (subsequent lines)

16 June 2011 Agile Mëtteg – Continuous Integration 45

SCM RULES FOR CONTINUOUS INTEGRATION

The idea is

to be able to easily link back a modification (with all its related changes) to an issue from the issue tracker

to be able to take out easily a modification

that should not be included in a specific release that breaks the build process that blocks the other developers that conflicts with other changes …

keep your code agile !

16 June 2011 Agile Mëtteg – Continuous Integration 46

List of tasks from issue

tracker[“assigned to

me”]

Details of a task

Activate a task

Filter: show only files worked on

for the currently active task

Modified files grouped by tasks

(“atomic” commits)

ISSUE TRACKER

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 55

AUTOMATED BUILD TOOLS

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 60

AUTOMATED BUILD TOOLS

16 June 2011 Agile Mëtteg – Continuous Integration 61

AUTOMATED BUILD TOOLS : REPOSITORIES

AUTOMATED BUILD TOOLS : REPOSITORIES

CONTINUOUS INTEGRATION SERVER

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 64

QUALITY ANALYSIS AND REPORTING

Tools

SCM

Issue Tracker

Automated Build

Continuous

Integration

Server

Quality Analysis

and Reportin

g

16 June 2011 Agile Mëtteg – Continuous Integration 67

QUALITY METRICS IN THE IDE

Agile Mëtteg – Continuous Integration 74

WRAP-UP

16 June 2011

Agile Mëtteg – Continuous Integration 75

CONCLUSIONS

Guaranteeing code quality is an intensive task• In time• In money

Continuous Integration pays back and offers a lot• Has a high ROI• Cost of ownership reduces over time

It can be applied incrementally

Agility should be the driving backbone for its adoption

16 June 2011

RESOURCES

Gartner’s study ID “G00151721” http://condor.depaul.edu/~dmumaugh/readings/handouts/SE477/Gartner%20Reports/from_the_cio_trenches_why_so_151721.pdf

Standish Group’s Chaos Reporthttp://www.standishgroup.com/services.php

“Quality Software Management : Systems Thinking”Gerald Weinberg, 1991, ISBN 978-0932633729

“Code Complete”, Microsoft Programming SeriesSteve McConnell, 1993, ISBN 978-1556154843

16 June 2011 Agile Mëtteg – Continuous Integration 76

RESOURCES

CVS http://www.nongnu.org/cvs/Subversion (SVN) http://subversion.apache.org/Maven http://maven.apache.org/Ant http://ant.apache.org/Nant http://nant.sourceforge.net/Ivy http://ant.apache.org/ivy/EasyAnt http://www.easyant.org/Gradle http://www.gradle.org/Buildr http://buildr.apache.org/Gant http://gant.codehaus.org/Jenkins CI http://jenkins-ci.org/Hudson CI http://hudson-ci.org/Sonar http://www.sonarsource.org/Microsoft Team Foundation Server http://www.microsoft.com/visualstudio/en-us/products/2010-editions/team-foundation-server/

16 June 2011 Agile Mëtteg – Continuous Integration 77

RESOURCES

Agile Partner: www.agilepartner.net

& Blog: http://blog.agilepartner.net

Trainingshttp://www.agilepartner.net/formations/coup-de-projecteur-sur/?lang=fr

Agile Interest Group LU: www.aiglu.orgAgile Tour Luxembourg8 November 2011

16 June 2011 Agile Mëtteg – Continuous Integration 78

CONTACTS

Thank You

16 June 2011 Agile Mëtteg – Continuous Integration 79

Jean-Pol LANDRAIN Sylvain CHERYConsultant Director

jplandrain@agilepartner.net

schery@agilepartner.net

+352 263 700 30 +352 691 555 221

DEBRIEFING

Questions ?

5 fingers vote

1 = useless“I gained nothing. I completely lost my time!”

2 = useful“It wasn’t worth all the time spent on it. I lost most of my time”

3 = average“I gained enough to justify the time spent on”

4 = above average“Good value, I gained more than the time spent”

5 = excellent“Really useful session, time well spent”

16 June 2011 Agile Mëtteg – Continuous Integration 80

Agile Mëtteg – Continuous Integration 81

EXTRAS

16 June 2011

BRANCH BY ABSTRACTION

“Branch by abstraction”

all the changes are done in the same place, same location in the SCM

no need to merge (hazardously) a lot of code from the feature branch

history of the changes stays easy to follow

16 June 2011 Agile Mëtteg – Continuous Integration 82

BRANCH BY ABSTRACTION

“Branch by abstraction”

Cookbook:

Introduce an abstraction over the core bits of the big thing you are going to change Update all the bits of code that were formerly using the thing directly to use it via the new abstraction

Make a second implementation of the abstraction, with unit tests that specifically test its core functionality

Update all the code to use the new implementation

Deprecate the first implementation

Delete the first implementation (there is no need to go back)

Remove the abstraction (only if it is inelegant, not often the case)

16 June 2011 Agile Mëtteg – Continuous Integration 83