how automation reveals technical debt

© 2013 IBM Corporation

How Automation Reveals Technical Debt

© 2013 IBM Corporation

Eric’s BioI’m a DevOps Evangelist at UrbanCode where I helps customers get the most out of their build, deploy and release processes.

I have automation experience throughout the application life-cycle in roles as a developer, test automation engineer, and support engineer. For the last 9 years, I’ve been focused on CI, CD and DevOps

Eric Minickeric@urbancode.com@EricMinick

© 2013 IBM Corporation

Technical Debt

© 2013 IBM Corporation

Why do we accumulate technical debt?

We leverage technical debt to deliver more faster.

De-leveraging is rarely accounted for in project planning.

Green-Shifting*

Scope Time

Resources

* http://www.drdobbs.com/191600661

© 2013 IBM Corporation

Why do we care? Paying interest

Now Later Later Still Much Later0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Debt PaidInterest PaidValue Delivered

Debt

© 2013 IBM Corporation

Why do we care? Or paying our debts

Now Later Later Still Much Later0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Debt PaidInterest PaidValue Delivered

Debt

© 2013 IBM Corporation

Pay it now, or pay more later

NowLa

ter

Late

r Still

Muc

h La

ter

Debt Paid

Interest Paid

Value De-livered

NowLa

ter

Late

r Still

Muc

h La

ter

Debt Paid

Interest Paid

Value De-livered

© 2013 IBM Corporation

Why should we care?

Baggage that slows the team

Lack of automated tests lengthen QA cycles

Fear of merging work

Unrefactored code slow to work with

Slow build / deploy processes delay learning and release pace

Quality issues

Lack of tests results in buggier code

Releases are error prone and lead to unnecessary outages

© 2013 IBM Corporation

The limits of what we know

Known Knowns: Bugs confirmed and tracked

Known Unknowns: Undiscovered bugs

Unknown unknowns: One of our project teams is using a GPL’d library making their product impossible to ship

© 2013 IBM Corporation

Where Automation Helps

“Testing” for debt: Automated tests, code scans and reports can help identify (and quantify) problems.

Automation as a learning experience: The act of automating brings surprises.

© 2013 IBM Corporation

Testing for Debt

Continuous Integration (multi-component)

Code Inspection

Watching Trends

© 2013 IBM Corporation

Testing for Debt: Continuous Integration

On code commit, build and test the software

Roll up changes to build-time or runtime dependencies and test those too to identify API incompatibilities

© 2013 IBM Corporation

Testing for Debt: Code Inspection

Code Reviews – Rapidly detect issues

Static Code Analysis – Tool based checks for bugs, code duplication, code theft, & non-compliance with dev standards.

© 2013 IBM Corporation

Testing for Debt: Code Inspection

Code Reviews – Rapidly detect issues

Static Code Analysis – Tool based checks for bugs, code duplication, code theft, & non-compliance with dev standards.

© 2013 IBM Corporation

Testing for Debt: Watching Trends

© 2013 IBM Corporation

More visualizations: Sonar

http://nemo.sonarsource.org/dashboard/index/327690?did=6

© 2013 IBM Corporation

An example safety net

Continuous build & unit test

Nightly slow tests / code scans

Emails identifying new issues – ideally tied against source code changes

Regular inspection of trends

Bugs / Stories entered around issues

© 2013 IBM Corporation

Automation as a learning experience

Implementing the safety net helps us discover unacknowledged debt

© 2013 IBM Corporation

Tests? What tests?

© 2013 IBM Corporation

Automation Examples: The Build

Build time dependencies not understood

Build scripts missing or incomplete

“Magic build server” anti-pattern

http://www.urbancode.com/html/resources/webinars/Role_of_Binary_repositories_in_Software_Configuration_Management.html

© 2013 IBM Corporation

Automation Examples: Deployment

Deployment scripts scarce

“Special Instructions” with most deployments indicate non-repeatable process

Environmental differences handled poorly

Separation of duties less than real

© 2013 IBM Corporation

Automation Example: Test Automation

Requirements less well understood

Existing tests are few, stale, un-optimized

Application not architected for testability

© 2013 IBM Corporation

Expect the unexpected when automating

At scale, Green-Shifting, has hidden issues

Include these discoveries in ROI estimations for automation (positively and negatively)

This is a happy side effect

© 2013 IBM Corporation

Start making decisions

© 2013 IBM Corporation

Direct and indirect automation benefits

Direct: We tested for problems and found them.

Indirect: In attempting to be more efficient, we automated, and accidently discovered problems.

© 2013 IBM Corporation

Automating for the team

Provide a “safety net” to detect and recognize issues.

Diagnose and repair lack of repeatability in build-deploy-test

Quantify accumulating debt in support of fighting scope creep

Team level tooling is fine

© 2013 IBM Corporation

Automating the enterprise

Benefits extend beyond aggregate team level benefits

Central Automation and Reporting gets us:

–Identify who can use shared configuration

–Who has tests, who doesn’t

–Who is using what tools

–Build / deploy failure rates

© 2013 IBM Corporation

Stories from customers

© 2013 IBM Corporation

Favorite Examples: Deployment Failures

Debt: High rate of deployment failures a problem

Interest: QA productivity is getting hurt & lengthened time to market

Goal: Reduce failure rate from 40% to 5%

Approach: Avoid manually fixing a deployment. Fix the automation and redeploy.

Enforcement: Monthly / weekly spreadsheet of success to CIO with a six month plan.

© 2013 IBM Corporation

Favorite Examples: “End of Day” Commits

Debt: Developers commit breaking changes at the end of the day

Interest: Code base broken in morning, or other people stay up late to fix it.

Goal: Avoid other devs staying late to clean up

Approach: Report on failures, and react to negative patterns as a team.

Enforcement: Social pressures

© 2013 IBM Corporation

© 2013 IBM Corporation

Favorite Examples: Low numbers of tests

Problem: Unit testing discipline breaking down over time

Goal: Maintain high or improving coverage

Approach: Standard coverage tools and an emphasis on upward trends.

Enforcement: Trending report

on monitor over CIO’s door

© 2013 IBM Corporation

No hiding!No greenshifting

© 2013 IBM Corporation

Summary

We accumulate technical debt as we race to deliver more, faster.

This causes us to eventually release less, slower, with worse quality

Automation directly and indirectly helps us find issues.

There are benefits at both team and enterprise levels.

© 2013 IBM Corporation

More References

http://urbancode.com/resources

Enterprise CD Maturity Model

Lean Build & Deployment Automation

Managing Release Risks with Metrics

Blogs.urbancode.com

Twitter.com/UrbanCode

facebook.com/IBMUrbanCodeProduc

Slideshare.net/Urbancode

© 2013 IBM Corporation

Yes, we sell products for this

uBuild

–Build management and continuous integration

uDeploy

–Deployment and release automation

uRelease

–Release management tool for planning and executing big releases

... And IBM’s amazing portfolio of CLM, testing tools, service virtualization, provisioning, etc, etc.

© 2013 IBM Corporation

Questions?

eminick@us.ibm.com@EricMinick, @UrbancodeSlideshare.net/Urbancode