devtalk: the road to continuous delivery: driving lessons
TRANSCRIPT
The Road to Continuous Delivery: Driving Lessons
Laurette Cisneros Engineering Tools ManagerPerforce
John WillistonProduct Marketing ManagerPerforce
The Road to Continuous Delivery: Driving Lessons
About the Presenters
2
John WillistonProduct Marketing Manager, Perforce
• A veteran software developer for Windows, .NET, and the web
• In his spare time: a musician, philosopher, gamer, and all around self-admitted geek, who spends much of his time bringing those skills to bear on the challenges of marketing.
The Road to Continuous Delivery: Driving Lessons
About the Presenters
3
Laurette Cisneros Engineering Tools Manager, Perforce
• 30 years experience in the software industry
• Extensive experience in release engineering, version control, and product development.
• In her spare time: Amateur enologist (and I do sample my own creations) and thinks Portal is the only game in town
The Road to Continuous Delivery: Driving Lessons
Perforce Overview
• Fastest, most scalable,version management and collaboration
• Commonly used forall types of content
– Code
– Binaries
– Movies
– Chip Designs
– Gaming
– Images
Global Availability and Support
The Road to Continuous Delivery: Driving Lessons
Agenda
5
• Why continuous delivery
• Our approach
• Lessons learned
The Road to Continuous Delivery: Driving Lessons
5 Years Ago – Before DevOps
• Manual builds and pushes operated by Build & Release team
• Machines and “sliders”
• Nightly builds: – cron, bash/perl
– Limited set of products/platforms
– Late night run, full day of changes
• “build-build” scripts– Beginnings of an abstraction layer
– Consistent interface for underlying “make” tool (jam)
6
The Road to Continuous Delivery: Driving Lessons
Business Priorities Required Change
7
• More products, platforms, programming languages
• Fighting priorities, limited resources
• Needed faster feedback on product changes
• Manual handoffs between teams causing delays
The Road to Continuous Delivery: Driving Lessons
Agenda
• Why continuous delivery
• Our approach
• Lessons learnt
8
The Road to Continuous Delivery: Driving Lessons
Our Approach
• Shared self-service release management infrastructure
• Transitioned build/test/deploy knowledge to product teams
• Trunk-based development and feature toggles
• Extensive automated testing with QA focused on exploratory testing
• Automatic (gated) releases
• Engineering Tools team
9
The Road to Continuous Delivery: Driving Lessons
Continuous Delivery Pipelines
10
DEV QA STGPERF PROD
Web Apps
DEV QA PERF
Packaged Apps
ProductManager
The Road to Continuous Delivery: Driving Lessons
Continuous Delivery Tool Chain
11
The Road to Continuous Delivery: Driving Lessons
An Example: From Submit to RPM
12
The Road to Continuous Delivery: Driving Lessons
Data Driven Improvements
• Gather Metrics– Via framework, insert into DB
– Wait time for resource
– Elapsed time on resource
• Identify bottlenecks
• Adjust
• See effects
13
The Road to Continuous Delivery: Driving Lessons
Data Driven Improvements
14
Automation
The Road to Continuous Delivery: Driving Lessons
Data Driven Improvements
15
The Road to Continuous Delivery: Driving Lessons
Results So Far
• Release process lead time down from 7 days to 4 hours
• Up to 75% automated test coverage– Releasing without regressions
• Massive increase in production releases– 450 releases in 2014
– 8 releases in 2012
– 19 releases in 2013
• Engineering Tools dedicated to strategic projects
16
The Road to Continuous Delivery: Driving Lessons
Agenda
• Why continuous delivery
• Our approach
• Lessons learned
17
An Insider View
18
Methodology
• Trunk based development
• Feature toggles
• Versioning everything
• Source
• Artwork
• Binaries
• VM Images
• Pragmatic approach, continuous improvement
Culture
• DevOps and CD opened doors internally
• Allowed increased infrastructure investment
• Increased empathy with developers
• Management education helped
• It’s not all about developers
• Set goals, not how to achieve them
Technology
• No such thing as a single “DevOps Solution”
• Some overlap in functionality in tools used - that’s okay
• Use the right tool for the job
• Mixing commercial and open source software has challenges but is essential
• Focus on what we can do today
The Road to Continuous Delivery: Driving Lessons
What’s Been Difficult
• Systems designed to be managed by a small group of experts– Transition to embedded team too abrupt
– Skills gap
• Teams understanding why shared infrastructure was necessary
• Instead of being responsible as an embedded team, just allocated tasks to an individual
• Ad-hoc requests from developers
19
The Road to Continuous Delivery: Driving Lessons
Continuous Improvements
• Stay on course
• Continuously measure impact
• Re-evaluate what’s been done and improve
• Continue to tackle technical debt
• Automated infrastructure testing
• Communication
20
The Road to Continuous Delivery: Driving Lessons
Questions?
John Williston: [email protected]
Laurette Cisneros: [email protected]
P4Ideax Forums