continuous delivery -...
TRANSCRIPT
Continuous Delivery Benefits, Best Practices and Practical Advice
Mark Warren
Perforce Software
Jeffrey Hammond
Forrester Research Ajit Zadgaonkar
Edmunds.com
Continuous Delivery: A Key Enabler
of Feedback and Flow
Jeffrey Hammond, VP, Principal Analyst
Sept. 24, 2013
2 Entire contents © 2010 Forrester Research, Inc. All rights reserved.
How Would You
Describe Your Release
Management Process?
2 Entire contents © 2010 Forrester Research, Inc. All rights reserved. Source: Flickr (http://bit.ly/13haRZ6/)
© 2013 Forrester Research, Inc. Reproduction Prohibited 3
Developers Employ Agile Practices…
Source: Forrsights Developer Survey, Q1 2013
Which of the following project management practices does your development team currently use?
19%
3%
3%
5%
6%
8%
8%
13%
12%
15%
20%
24%
28%
29%
35%
None of the above
Minimum viable product approach used to focus releases
A/B testing used to evaluate proposed changes
Kanban board used to show progress
Stage-gate project approvals
Code coverage threshold must be met for release
Pair programming
Use a burn-down list to prioritize day-to-day activity
Run retrospectives at the end of each development phase
Use visualization, visual prototyping, storyboarding, or wire-…
Product owner involved in development
Daily standup/Scrum meetings
Test-driven development
Short development iterations
Formal code review
Base: 698 Professional Developers, Game Developers, IT Developers, Consultants
© 2013 Forrester Research, Inc. Reproduction Prohibited 4
… But Spend More Time On E-Mail Than Deploying Code or Writing Tests
Source: Forrsights Developer Survey, Q1 2013
How much time do you spend in a typical work day doing the following activities?
65%
52% 50% 49%
38% 35%
26% 23% 19%
14% 12%
Base: 651 Professional Developers, Game Developers, IT Developers, Consultants
% of Developer That Spend More Than an Hour A Day
© 2013 Forrester Research, Inc. Reproduction Prohibited 5 Source: Flickr (http://bit.ly/13R9WgS/)
This type of thinking is becoming increasingly
obsolete…
6 Entire contents © 2010 Forrester Research, Inc. All rights reserved.
Source: Flickr (http://www.flickr.com/photos/sashawolff/3793206523/sizes/l/)
9/11 1.3 M+ Android activations per day, 500 million+ total devices
9/12 400 M+ iOS devices sold to date
Over 700M smart phones sold worldwide in 2012
Samsung Galaxy SIII: 96 million, iPhone 5: 54 million
The Mobile Shift is upon us…
© 2013 Forrester Research, Inc. Reproduction Prohibited 7
IT Shops Struggle To Keep Up With Mobile
Source: Forrsights Developer Survey, Q1 2013
How often does your development team release mobile apps?
21%
28%
11%
11%
12%
13%
4%
25%
39%
13%
2%
8%
10%
2%
0% 20% 40% 60%
<=1 release/yr/device
2-4 regular releases/yr/device
5+ regular releases/yr/device
Every month or more (as needed)
Releases to coincide with new versions ofdevice OS
We release patches for defects as needed andthen release major updates when we are ready
Other (please describe)
IT Dev
Pro Dev
Base: 172 Professional software developers and 87 IT developers building mobile applications
© 2013 Forrester Research, Inc. Reproduction Prohibited 8
Mobile apps are just part of modern applications
The way we develop is changing…
7 Traits of Modern Applications
1. Omni-channel clients
2. Deployed on elastic
infrastructure
3. Aggregate discrete services
4. Use managed APIs
5. Integrate open source
software
6. Employ dev-ops techniques
7. Focus on measurable
feedback
Systems of
Engagement
Delivering modern applications is hard
Time to Safety
Time to Certainty
Time to Feedback
Systems of
Operation
Systems of
Record
Lifecycle Focus
Getting real customer feedback is hard
without frequent production releases!
Continuous Delivery Enables Fast Feedback
Flow
What’s your “Null Release” cycle time?
What is the “Null Release” ?
“If we changed one line of code in our application
(or system), how long would it take us to deploy
it into production using our regular release
process?”
5 ways to streamline delivery
1. Improve pre-build processes
2. Expand release management throughput
3. Optimize your release pipeline
4. Architect software for rapid change
5. Create a common release portal
Improving pre-build processes
1. Outlaw private builds
and release streams
2. Employ gated check-
ins and pre-flight
builds
3. Make your continuous
integration process
change aware
Garbage in, garbage out
Expanding release throughput
X X X X
Sprint
2
Sprint
1
Sprint
3
Sprint
4
Sprint
6
Sprint
5
Sprint
7
Sprint
8
1. Apply the concept of a release
pipeline to your work.
2. Use "smoke tests" to evaluate
builds quickly
3. Decompose each process into
the smallest possible logically
independent steps
4. Automate repetitive steps
wherever possible
5. Use parallelism to speed up
builds
Adopt A Simple Release Pipeline
Optimize your delivery pipeline
› Use virtual images to enforce consistency
› Use parallelism to compress the testing phase
› Use masked production data to improve testing
fidelity
› Implement a developer self-service
environment
› Use configuration management to automate
deployment
› Use service virtualization to speed testing
An Optimized Delivery Pipeline
Architect Software For Rapid Change
› Modularize your architecture
› Manage deployment by configuration
› Design systems to support "blue-green"
deployment
› Use runtime configurations of features to avoid
rollbacks
› Web services, message passing to decouple
layers
Create A Common Release Portal
What you need
1. Status of the release
pipeline
2. Drilldown metrics on
individual releases
3. Consistent locations and
links for "latest releases.
4. Links to all configuration
recipes and instances
5. Alerting
© 2012 Forrester Research, Inc. Reproduction Prohibited
Tools help increase your rate of delivery
22
Release Planning
Release Automation and Orchestration
Work Item Management and Tracking
Continuous Integration and
Build Management
Service
Management
Configuration Management
Test Management And Execution
SCM
Portfolio Management
Monitoring
and APM
Systems of
Engagement
The “need for speed” may vary
Milestone builds
Release on demand
Hypothesis driven
Systems of
Operation
Systems of
Record
© 2013 Forrester Research, Inc. Reproduction Prohibited 24
When release management works
› Amazon – a release every 11.6 seconds, leading
public cloud infrastructure
› Netflix – testing in production w/ the Simian
army, ~30% of NA internet traffic
› Instagram – 0 – 14 million customers in 1 year,
massive wealth creation for the dev team
› Eclipse foundation – 50M LOC shipped like
clockwork, every year for 6+ years.
© 2013 Forrester Research, Inc. Reproduction Prohibited 25
Next steps
1. Make sure you are planning for faster releases
as part of your modern application strategy
2. Assess your current level of maturity, and what
it takes to get to the next level
3. Understand the cultural changes required to
get to a state of continuous delivery
4. Tools and practices will help reinforce cultural
change
5. Segment opportunities by complexity and
experience and act accordingly
Continuous Delivery The Role of Version Management
Mark Warren
Product Strategy Director
Perforce Software
• Headquartered in Alameda,
California, international operations
in the United Kingdom, Canada
and Australia.
• Trusted with storing and versioning
the most valuable IP for the world's
most innovative companies
– Over 10,000 customers worldwide
– From start-ups to Fortune 100
– The leading cloud companies
Perforce Software
CD Enabler – Version Management
SCM
De
ve
lop
me
nt
Bu
ild
& T
es
t
Re
lea
se
Op
era
tio
ns
& M
on
ito
rin
g
Version Everything
.doc .jpg
.mp4 .svg
Build
Script Libraries
DB
Script Compilers
Installer Env.
Config
.java .exe/.dll
.war .db
Java C#
.jsp
COBOL
CSS
Ruby
PHP
Python
.xls .docx
.proj
Not Just Software Developers
• Documenters, authors, artists,
project managers, release
managers, Natural
tools/UI/workflows
All Teams and Projects
• Wherever they are
Version Everything – CCP Games
• Version absolutely *everything*
• Branching – “where it makes sense”
• “Nobody is born a versioning hero”
• “Versioning everything requires an
investment. It’s worth it.”
Versioning is the nerve
center of the organization { }
Mainline Release Branching Strategy
Release 1.1.x
Release 1.0.x R R R
R R
merge
bugfix
merge
bugfix
• Visibility
• Always deployable
• Continuous Integration, only create branches for releases
• Manage what gets into mainline
Mainline
Mainline – Salesforce.com
• 150 teams, 5,000 testing VMs, 500,000 files.
• Up to 10 million Perforce transactions per day,
heading towards 15 – 20 million
• Mainline code model
• Fast feedback is key
• Security, audit, compliance. Easy audit process
• “Perforce is boring” – For ops team, that’s a very
good thing!
Mainline is critical to collaboration { }
Best Practices
• Version *everything*
• Automate everything
• Love the Operations team
• Invest in infrastructure
• Feedback early & often
• Used by everyone
For more information
For more use cases and information on
the role of version management…
perforce.com
Continuous Delivery
Copyright Edmunds.com, Inc. (the “Company”). Edmunds and the Edmunds.com logo are registered
trademarks of the Company. This document contains proprietary and/or confidential information of the
Company. No part of this document or the information it contains may be used, or disclosed to any
person or entity, for any purpose other than advancing the best interests of the Company, and any such
disclosure requires the express approval of the Company.
Ajit Zadgaonkar Sr Director, Software Engineering
Sept 24, 2013
Edmunds.com
• 170M monthly page views
• 13M monthly unique visitors
• 500 Employees
• Externally Exposed APIs: 175+ apps /month
• 1300+ active developers
Software Delivery Pitfalls
• Longer shelf life of features waiting
for releases
• Lower code readiness confidence
• Recurring code integration issues
• Release qualifications & promotions
were ~ 90+man days
• Business was dependent on
release schedules
Need for CD
We wanted to…
• Be more agile, lean and nimble towards
delivering features to our consumers
• Operate in a proactive and faster manner
• Not compromise quality, stability and
reliability.
Transformation Path
• Only mainline code branch
• Ensure backwards and forwards compatibility
• Keep code in good state
• Avoid code merge
• Reliable, stable automated tests and monitoring
• Automated deployments
The Difficult
Challenges
• Doing it right
• Automate for scalability and flexibility
• Employ tools – to gain efficiency
• Transition to adoption
• Having automated tests result into higher confidence
• Code compatibility
• Accepting feedback
• Everything is code • Code need to be production ready all the time
• Remove conflicting or stale dependencies
• Clean up redundant code (feature, dependencies,
code, monitors)
• Let test results define code/feature readiness
• Time is of the essence • Faster running code, sooner feedback
• Build time, test execution time
• Immediate feedback is key - delay is build break!
• Fix-it-now (drop everything) culture • Fixing issues in our work is part of the same day,
same iteration
Critical Success Factors
Critical Success Factors
• Feedback matters, not the mechanism
• Must be followed up with immediate actions i.e.
• Code corrections
• Tests added / fixed / removed
• Deployments scripts fixed, Application
configuration updated
• Feedback Accountability
• Alerts / Monitors are for taking actions
• Feedback improves us, You will get feedback when
you are open
• You can’t improve what you don’t measure
• No tests, no benchmark,
no monitors = no feedback
Workflow
Checkin > static code scanning > unit Tests > code
coverage > build (parallel) > repository > CD invoke
> deploy (parallel)> post deployment checks >smoke
tests > regression tests > compatibility tests > Load
Performance tests > metric monitors > promote >
smoke test > monitor
Check-in Static code
scanning unit tests
code coverage
build(parallel) repository CD
invoke
deploy(parallel) Post deployment
checks
smoke
tests
regression tests
Metric
monitors promote
Smoke
test
monitor
Workflow
Compatibility
tests Load Performance tests
Overview: Environment
CI
DEV QA PROD
Build
Deploy Deploy Deploy
Test/Monitoring
Watch
Outside view: “it’s automation”
Inside view: “it’s confidence”
Continuous Delivery
We did it, you can too
