Build & Deployment InitiativePerformance Analysis Summary
Agenda• Overview
• Performance Analysis
▫ Data Gathering
▫ Root Cause Analysis
• Interventions
▫ Selection & Evaluation
▫ Justification
▫ Impact
• Analysis Summary
• Q&A
Overview
Opportunities
• Enable continuous integration builds
• Automatic bug creation for failures
• Triage team to perform root cause analysis on recurring failures
Benefits
• Developer productivity increased
• Test productivity increased
Key DefinitionsBuilds: custom code that is written by our developers and compiled into packages that can be deployed.
Deploys: the installation of compiled code packages onto specific servers within the websites infrastructure.
LOB: Line of Business (different business/product groups).
Problem
• Average 70% Build & Deploy success rate over 6 months prior to Intervention
• All Corp and LOB applications are below the 95% target success rate
Impact
• Developer productivity decreased;
• Tester productivity decreased
Performance Analysis
Goal
• Determine root cause of low build and deploy success rates.
Steps
• Data Gathering
• Root Cause Analysis
Data GatheringData from Tools
• Build success metrics
• Bugs logged against projects
• Tracked impacting issues
Data from Employees
• Processes and common issues
• Schedule and workload impacts
• Improvement ideas and suggestions
RAID
•Tracks build success and failure rates, duration, group, project, and release.
•Can create reports from this data, showing success rates across all projects and frequency of failures.
Anthill Pro
•Tracks all bugs found during a project, including build bugs.
•All bugs are tracking here and can be queried for build failure bugs. Typically includes a great level of detail.
SharePoint
•Tracks the top issues within the entire organization across all disciplines and groups.
•May contain non-bug related problems with the build process or higher level trends resulting from low build success rates.
Build Engineers
•Responsible for running the builds and troubleshooting any failures. Firsthand knowledge of common issues and trends across the projects.
Developers
•Write the code that is compiled and familiar with code related build failures and pre-code-checkinprocesses.
Dev Leads
•Responsible for code quality and processes developers must follow to deliver code. Will be familiar with code quality trends and the impact on development from low build success rates.
Project Managers
•Run the projects and track all issues and risks related to the project. Will be familiar with schedule impacts as a result of low build success rates.
Testers
•Test the builds once they have been deployed to test environments and will be familiar with issues blocking testing that may be related to the build and deploy process.
Data Sources: Tools
Data Sources: Employees
Methods Used
• Database queries
• Surveys
• Interviews
Root Cause Analysis
Problem
• Average 70% Build & Deploy success rate over last 6 months
• All Corp and LOB applications are below the 95% target success rate
Impact
• Developer productivity decreased
• Tester productivity decreased
Ro
ot
Ca
use
Disparate Processes: *Different LOB applications use different build and deployment
processes, bug creation standards methods, and issue analysis.
LOB Specific Tools: *40% of teams use non-standard build and deploy tools, increasing
tool support and licensing costs.
No Pre-build Validation: *Modified code is not validated for build errors prior to the scheduled
full build.
There are 3 main root causes that have contributed to the low build and deploy success rate:
Interventions
Goal
• Identify and implement specific actions to improve the low build and deploy success rate
Steps
• Selection & Evaluation
• Justification
• Impact
Selection & Evaluation
There are three main areas of improvement that each have a targeted intervention.
Interventions are evaluated based on five criteria:
Evaluation Criteria
• Specific
• Measureable
• Action-oriented
• Results-oriented
• Trackable
Intervention Justification
Organizational Goals
Quick delivery of new
features
High quality releases
Intervention Impact
Immediate identification of build breaks
Improves tracking of bugs and issues
Systemic issues identified and resolved
CI Builds
Automatic Bug Creation
Root Cause Analysis
Direct link between interventions and organizational goals.
One-time changes for build and deployment technology and processes.
Improvement in build and deploy success rate improves time to market and quality of releases.
Intervention Impact
Improvements
• Average 91% Build & Deploy success rate over 2 months following the Intervention
• All Corp and LOB applications are trending upwards towards 95% build and deploy success rate
• Bugs are created automatically upon any build or deploy failures
• Root cause for systemic problems is reducing problem recurrence, improving build and deploy success rates
Analysis Summary
December*Builds and deploys at a 70% success rate
January*Interventions implemented*Immediate improvement seen*Success rates at 82% and trending upwards
February*Success rates at 91% and trending upwards
March*Expected build and deploy success rates at or above 95% goal
Problem: Low build and deploy success rate
Interventions:
• Continuous integration builds
• Root cause analysis of failures
• Auto bug creation
Q&A