Download - Agile Mëtteg - June 2011
Software Dev. Practices – Continuous Integration
Agile Mëtteg – June 16th, 2011
Agile Mëtteg – Continuous Integration 2
ABOUT US
16 June 2011
PROFILE
Created in 2004Independent Software Development Company
16 June 2011 Agile Mëtteg – Continuous Integration 3
FIGURES
16 June 2011 Agile Mëtteg – Continuous Integration 4
2004 2005 2006 2007 2008 2009 20100 €
500,000 €
1,000,000 €
1,500,000 €
2,000,000 €
2,500,000 €
3,000,000 €
Turnover
16%
1%
42%
41%
Turnover Distribution
Place FinancièreActivités IndustriellesServices & VenteSecteur public & Associatif
2,5M€ en 2010
2004 2005 2006 2007 2008 2009 2010 2011-5
0
5
10
15
20
25
30
35
0 -1 0 -1 0 -2 -1 00 29 12 13 16
2227
2
8
32 3
8
6
5
Employees turnover
32 in 2011
MISSION
Design, Develop and Customize “Software as Business & Operational Enabler”
Fast & flexible solutions business value oriented
Help IT and all Business & Operational Organizations to adopt the culture of “Software as Business & Operational Enabler”
Simple & pragmatic methods making effective the collaboration of the actors of a projectEasy and powerful tools for follow-up of relevant KPI
16 June 2011 Agile Mëtteg – Continuous Integration 5
OUR SERVICES
MgtTeam
Services
SoftwareDevelopme
nt
OpsTeam
Services
Dev Team
Services
123
4
Applications enabling productivity
Bespoke application
Mobile application
Based on package
Consulting services Coaching & SupportTrainingResource delegation
16 June 2011 Agile Mëtteg – Continuous Integration 6
1
2
3
4
OUR MEANS
16 June 2011 Agile Mëtteg – Continuous Integration 7
80% > 4 years56% > 8 years
31% > 12 years
Agility
Authorized Training Center in Luxembourg
0-4 4-8 8-12 12-16 16- 0
2
4
6
8
10
5 68
64
Overall seniority
OUR MAIN CUSTOMERS
16 June 2011 Agile Mëtteg – Continuous Integration 8
Agile Mëtteg – Continuous Integration 9
SPEAKERS
16 June 2011
Jean-Pol LANDRAIN
Sylvain CHERY
Consultant DirectorCSM - CSP
Agile Mëtteg – Continuous Integration 10
ABOUT YOU
16 June 2011
Agile Mëtteg – Continuous Integration 11
PARTICIPANTS
Who are you ?
What is your role ?
What do you know about agility ?
16 June 2011
Agile Mëtteg – Continuous Integration 12
INTRODUCTION
16 June 2011
CONTINUOUS INTEGRATION
Continuous Integration in a few questions
Why do I need it ?
What is it ?
What does it require ?
How does it relate to the Agile principles ?
16 June 2011 Agile Mëtteg – Continuous Integration 13
SOFTWARE DEVELOPMENT: INDUSTRY? PROCESSES?
What is Software Development?
an industry mining, farming, construction, manufacturing,
…etc.
with clearly identified and well-defined processes, i.e. easily reproducible
Really?No failed project?A lot of…
16 June 2011 Agile Mëtteg – Continuous Integration 14
SOFTWARE DEVELOPMENT: A LOT OF FAILED PROJECTS
Gartner
75% of failure on at least one of
Cost Quality Time
16 June 2011 Agile Mëtteg – Continuous Integration 15
SOFTWARE DEVELOPMENT: A LOT OF FAILED PROJECTS
Source Result
KPMG Canada – Survey 1997
61% of failure, out of which:
• 30% over time• 50% tremendously over
budget
Standish Group (US) – Survey 2008
16,2% of success in all aspects:
• cost• time• deliverables
« Improbable success » in 68% of the cases
OASIG Study (UK) 1995 70% of failure « in some respect »
16 June 2011 Agile Mëtteg – Continuous Integration 16
« Houston, we’ve got a problem! »
16 June 2011 Agile Mëtteg – Continuous Integration 17
AGILE MANIFESTO vs SOFTWARE INDUSTRIALIZATION ?
Agile Manifesto
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
16 June 2011 Agile Mëtteg – Continuous Integration 18
SOFTWARE CRAFTSMANSHIP vs SOFTWARE INDUSTRIALIZATION?
Manifesto for Software CraftsmanshipNot only working software, but also
well-crafted softwareNot only responding to change, but also
steadily adding valueNot only individuals and interactions, but also
a community of professionalsNot only customer collaboration, but also
productive partnerships
16 June 2011 Agile Mëtteg – Continuous Integration 19
SOFTWARE DEVELOPMENT: AN ART?
Agile Manifesto and Manifesto for Software Craftsmanship were created by veterans of the software industry
However, when summed up, one can conclude software development is more an art than an industrial process
So, let’s compare with music…16 June 2011 Agile Mëtteg – Continuous Integration 20
SOFTWARE DEVELOPMENT: AN ART?
16 June 2011 Agile Mëtteg – Continuous Integration 21
SOFTWARE DEVELOPMENT: AN ART?
16 June 2011 Agile Mëtteg – Continuous Integration 22
SOFTWARE DEVELOPMENT IS AN ART
So, Software Development is an art!
But building nearly anything is also an art
Don’t you think?
And, more importantly,
Art is not without rules and best practices
16 June 2011 Agile Mëtteg – Continuous Integration 23
RULES AND BEST PRACTICES
What’s the worst?
Not following the rules usually leads to a rather direct and abrupt failure
• project fails integration testing • project is refused by infrastructure• project fails user acceptance testing• …
16 June 2011 Agile Mëtteg – Continuous Integration 24
RULES AND BEST PRACTICES
What’s the worst?
Not following the best practices augments, sometimes dramatically, the risks of failure
• reduces overall quality• increases the time to market / time to
deliver• has typically a pervasive effect: can be
ignored or remains unknown until it becomes really critical
• increases the maintenance and ownership costs
16 June 2011 Agile Mëtteg – Continuous Integration 25
RULES AND BEST PRACTICES
What’s the worst?
Both are terrible: sources of failureBut one is harder to detect than the other
Necessity to put in place and to define the structures and infrastructures required to check the quality at every level
Because “the earlier, the best” (and less expensive)
16 June 2011 Agile Mëtteg – Continuous Integration 26
Agile Mëtteg – Continuous Integration 27
SOFTWARE QUALITY
16 June 2011
WHAT IS SOFTWARE QUALITY?
“Quality is value to some person” (Gerald Weinberg, “Quality Software Management”)
i.e. quality is inherently subjective; people experience the quality of the same software very differently
this applies mainly to the quality of a software product, as perceived from an external view; and it can also comprise the quality of its running environment(s)
but the quality of the source code can also have an impact on the efforts needed for having a software product that fulfills the requirements, the intrinsic quality
Steve McConnell defines external and internal quality characteristics (‘”Code Complete”)
16 June 2011 Agile Mëtteg – Continuous Integration 28
IMPLICIT FACTORS FOR SOFTWARE QUALITY
The software projects typically follow rather detailed requirement plans
Though a set of characteristics often goes unmentioned
These are the implied requirements that are expected of all professionally developed software
16 June 2011 Agile Mëtteg – Continuous Integration 29
IMPLICIT FACTORS FOR SOFTWARE QUALITY
Conformance to implied requirements:
UnderstandabilityCompletenessConcisenessPortabilityMaintainabilityTestabilityUsabilityReliabilityEfficiencySecurity
16 June 2011 Agile Mëtteg – Continuous Integration 30
GUARANTEEING CODE QUALITY
How can we guarantee the level of quality of software during the « coding » phases?
The same ways as in the industry
• Factory inspections: “static code analysis” in IT
• Incremental improvements: “release soon, release often” in IT
• Tests, tests, tests, tests, tests, tests, …
16 June 2011 Agile Mëtteg – Continuous Integration 31
GUARANTEEING CODE QUALITY
Focus on Unit Tests and Test Driven Development (TDD)
Unit Tests
• The first tests to write• Written by the developers before the code• Write the code afterward to make the tests succeed• Must run in isolation, without any infrastructure• Must be fast to execute
How many unit tests per method ?
• One test per degree of “cyclomatic complexity”
16 June 2011 Agile Mëtteg – Continuous Integration 32
GUARANTEEING CODE QUALITY
Cyclomatic complexity of a method?
• the number of linearly independent paths in the method
If .. then If .. then .. else If .. and .. then If .. or .. then
Do .. While While .. Do Switch
16 June 2011 Agile Mëtteg – Continuous Integration 33
GUARANTEEING CODE QUALITY
Necessity to put in place the supporting
Structures (practices):
• Unit testing• Integration testing• Performance testing• Regression testing• Acceptance testing
Infrastructures (tools):
• Source Code management• Issue tracking• Build tools• Continuous Integration server• Quality reporting tools
Build Industrialization
Build IndustrializationPlatform
Ag
ility
Ag
ility
16 June 2011 Agile Mëtteg – Continuous Integration 34
GUARANTEEING CODE QUALITY
How to put in place the supporting
Structures (practices):
• Define and prioritize the quality requirements• Refine and adapt the relevant quality criteria to the objectives• Communicate rules and best practices: sharing, training and mentoring• Verify the correct usage and level of adoption• Review the results and improve the processes
Infrastructures (tools):
• Select the tools adapted to the defined quality requirements• Set them up based on the selected criteria (define measurements)• Communicate on their usage: sharing, training and mentoring• Generate quality reports and metrics• Analyze the conformance of the results and adapt the tools
Ag
ility
Ag
ility
16 June 2011 Agile Mëtteg – Continuous Integration 35
Agile Mëtteg – Continuous Integration 36
CI RULES AND PRE-REQUISITES
16 June 2011
CI RULES AND PRE-REQUISITES (1/3)
maintain a code repository
automate the build
the build must be self-testing
regular code sharingi.e. frequent commits
16 June 2011 Agile Mëtteg – Continuous Integration 37
CI RULES AND PRE-REQUISITES (2/3)
self-contained modificationsi.e. commits don’t break the build process
the build must be fast
integration tests (for the least) in a clone of the production environment
16 June 2011 Agile Mëtteg – Continuous Integration 38
CI RULES AND PRE-REQUISITES (3/3)
latest deliverables easily available to anyone who needs them
easy access to the results of the tests
automated deployments to a live test server
continous deployment to production is the ideal achievement
16 June 2011 Agile Mëtteg – Continuous Integration 39
Agile Mëtteg – Continuous Integration 40
BUILD INDUSTRIALIZATION PLATFORM
16 June 2011
BUILD INDUSTRIALIZATION PLATFORM
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 41
SOURCE CODE MANAGEMENT
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 42
SOURCE CODE MANAGEMENT
What are SCM tools ?
Tools that allow sharing and versioning files… but really not their primary interest …
… shared folders can do that too !
Tools to track and document the changes in the code, in such a way the developers can work in a task or
issue oriented mode
16 June 2011 Agile Mëtteg – Continuous Integration 43
SCM RULES FOR CONTINUOUS INTEGRATION
Tag your releases
Use branching strategies
Release branches
Feature branches for large changes
“branch by abstraction” pattern works well with continuous integration (for non-distributed SCM’s like CVS, SVN, TFS…)
16 June 2011 Agile Mëtteg – Continuous Integration 44
SCM RULES FOR CONTINUOUS INTEGRATION
Commit “atomically”
all the files related to a single task/issue in one operation or, if really not possible, max a few operations
don’t mix changes from different tasks/issues in the same commit
add a meaningful, standardized comment that (for example) includes:
• the task/issue identifier (first line)• status (first line)
« completed » or « in progress »
• short description from the issue tracking system (first line)• eventually a dedicated URL from the issue tracking system (second
line)• a meaningful description of what’s been done (subsequent lines)
16 June 2011 Agile Mëtteg – Continuous Integration 45
SCM RULES FOR CONTINUOUS INTEGRATION
The idea is
to be able to easily link back a modification (with all its related changes) to an issue from the issue tracker
to be able to take out easily a modification
that should not be included in a specific release that breaks the build process that blocks the other developers that conflicts with other changes …
keep your code agile !
16 June 2011 Agile Mëtteg – Continuous Integration 46
List of tasks from issue
tracker[“assigned to
me”]
Details of a task
Activate a task
Filter: show only files worked on
for the currently active task
Modified files grouped by tasks
(“atomic” commits)
ISSUE TRACKER
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 55
AUTOMATED BUILD TOOLS
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 60
AUTOMATED BUILD TOOLS
16 June 2011 Agile Mëtteg – Continuous Integration 61
AUTOMATED BUILD TOOLS : REPOSITORIES
AUTOMATED BUILD TOOLS : REPOSITORIES
CONTINUOUS INTEGRATION SERVER
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 64
QUALITY ANALYSIS AND REPORTING
Tools
SCM
Issue Tracker
Automated Build
Continuous
Integration
Server
Quality Analysis
and Reportin
g
16 June 2011 Agile Mëtteg – Continuous Integration 67
QUALITY METRICS IN THE IDE
Agile Mëtteg – Continuous Integration 74
WRAP-UP
16 June 2011
Agile Mëtteg – Continuous Integration 75
CONCLUSIONS
Guaranteeing code quality is an intensive task• In time• In money
Continuous Integration pays back and offers a lot• Has a high ROI• Cost of ownership reduces over time
It can be applied incrementally
Agility should be the driving backbone for its adoption
16 June 2011
RESOURCES
Gartner’s study ID “G00151721” http://condor.depaul.edu/~dmumaugh/readings/handouts/SE477/Gartner%20Reports/from_the_cio_trenches_why_so_151721.pdf
Standish Group’s Chaos Reporthttp://www.standishgroup.com/services.php
“Quality Software Management : Systems Thinking”Gerald Weinberg, 1991, ISBN 978-0932633729
“Code Complete”, Microsoft Programming SeriesSteve McConnell, 1993, ISBN 978-1556154843
16 June 2011 Agile Mëtteg – Continuous Integration 76
RESOURCES
CVS http://www.nongnu.org/cvs/Subversion (SVN) http://subversion.apache.org/Maven http://maven.apache.org/Ant http://ant.apache.org/Nant http://nant.sourceforge.net/Ivy http://ant.apache.org/ivy/EasyAnt http://www.easyant.org/Gradle http://www.gradle.org/Buildr http://buildr.apache.org/Gant http://gant.codehaus.org/Jenkins CI http://jenkins-ci.org/Hudson CI http://hudson-ci.org/Sonar http://www.sonarsource.org/Microsoft Team Foundation Server http://www.microsoft.com/visualstudio/en-us/products/2010-editions/team-foundation-server/
16 June 2011 Agile Mëtteg – Continuous Integration 77
RESOURCES
Agile Partner: www.agilepartner.net
& Blog: http://blog.agilepartner.net
Trainingshttp://www.agilepartner.net/formations/coup-de-projecteur-sur/?lang=fr
Agile Interest Group LU: www.aiglu.orgAgile Tour Luxembourg8 November 2011
16 June 2011 Agile Mëtteg – Continuous Integration 78
CONTACTS
Thank You
16 June 2011 Agile Mëtteg – Continuous Integration 79
Jean-Pol LANDRAIN Sylvain CHERYConsultant Director
+352 263 700 30 +352 691 555 221
DEBRIEFING
Questions ?
5 fingers vote
1 = useless“I gained nothing. I completely lost my time!”
2 = useful“It wasn’t worth all the time spent on it. I lost most of my time”
3 = average“I gained enough to justify the time spent on”
4 = above average“Good value, I gained more than the time spent”
5 = excellent“Really useful session, time well spent”
16 June 2011 Agile Mëtteg – Continuous Integration 80
Agile Mëtteg – Continuous Integration 81
EXTRAS
16 June 2011
BRANCH BY ABSTRACTION
“Branch by abstraction”
all the changes are done in the same place, same location in the SCM
no need to merge (hazardously) a lot of code from the feature branch
history of the changes stays easy to follow
16 June 2011 Agile Mëtteg – Continuous Integration 82
BRANCH BY ABSTRACTION
“Branch by abstraction”
Cookbook:
Introduce an abstraction over the core bits of the big thing you are going to change Update all the bits of code that were formerly using the thing directly to use it via the new abstraction
Make a second implementation of the abstraction, with unit tests that specifically test its core functionality
Update all the code to use the new implementation
Deprecate the first implementation
Delete the first implementation (there is no need to go back)
Remove the abstraction (only if it is inelegant, not often the case)
16 June 2011 Agile Mëtteg – Continuous Integration 83