Business Value ofAgile Methods

Benefits of Testing Early & OftenDr. David F. Rico, PMP, CSEP, ACP, CSM, SAFe

Twitter: @dr_david_f_ricoWebsite: http://www.davidfrico.com

LinkedIn: http://www.linkedin.com/in/davidfricoAgile Capabilities: http://davidfrico.com/rico-capability-agile.pdf

Agile Resources: http://www.davidfrico.com/daves-agile-resources.htmAgile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf

Author Background Gov’t contractor with 32+ years of IT experience B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys. Large gov’t projects in U.S., Far/Mid-East, & Europe

2

Career systems & software engineering methodologist Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000NASA, USAF, Navy, Army, DISA, & DARPA projects Published seven books & numerous journal articles Intn’l keynote speaker, 130 talks to 12,000+ people Specializes in metrics, models, & cost engineeringCloud Computing, SOA, Web Services, FOSS, etc. Adjunct at five Washington, DC-area universities

Today’s Whirlwind Environment

3

OverrunsAttritionEscalationRunawaysCancellation

GlobalCompetition

DemandingCustomers

OrganizationDownsizing

SystemComplexity

TechnologyChange

VagueRequirements

Work LifeImbalance

InefficiencyHigh O&MLower DoQVulnerableN-M Breach

ReducedIT Budgets

81 MonthCycle Times

RedundantData Centers

Lack ofInteroperability

PoorIT Security

OverburdeningLegacy Systems

ObsoleteTechnology & Skills

Pine, B. J. (1993). Mass customization: The new frontier in business competition. Boston, MA: Harvard Business School Press.Pontius, R. W. (2012). Acquisition of IT: Improving efficiency and effectiveness in IT acquisition in the DoD. Second Annual AFEI/NDIA Conference on Agile in DoD, Springfield, VA, USA.

Traditional Projects

4

Big projects result in poor quality and scope changes Productivity declines with long queues/wait times Large projects are unsuccessful or canceled

Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.

Size vs. Quality

DE

FEC

TS

0.00

3.20

6.40

9.60

12.80

16.00

0 2 6 25 100 400

SIZE

Size vs. Productivity

PR

OD

UC

TIV

ITY

0.00

1.00

2.00

3.00

4.00

5.00

0 2 6 25 100 400

SIZE

Size vs. Change

CH

AN

GE

0%

8%

16%

24%

32%

40%

0 2 6 25 100 400

SIZE

Size vs. SuccessS

UC

CE

SS

0%

12%

24%

36%

48%

60%

0 2 6 25 100 400

SIZE

Global Project Failures

5Standish Group. (2010). Chaos summary 2010. Boston, MA: Author.Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.

Challenged and failed projects hover at 67% Big projects fail more often, which is 5% to 10% Of $1.7T spent on IT projects, over $858B were lost

16% 53% 31%

27% 33% 40%

26% 46% 28%

28% 49% 23%

34% 51% 15%

29% 53% 18%

35% 46% 19%

32% 44% 24%

33% 41% 26%

0% 20% 40% 60% 80% 100%

1994

1996

1998

2000

2002

2004

2006

2008

2010

Year

Successful Challenged Failed

$0.0

$0.4

$0.7

$1.1

$1.4

$1.8

2002 2003 2004 2005 2006 2007 2008 2009 2010

Trill

ions

(US

Dolla

rs)

Expenditures Failed Investments

Requirements Defects & Waste

6Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.

Requirements defects are #1 reason projects fail Traditional projects specify too many requirements More than 65% of requirements are never used at all

Other 7%

Requirements47%

Design28%

Implementation18%

Defects

Always 7%

Often 13%

Sometimes16%

Rarely19%

Never45%

Waste

What is Agility? A-gil-i-ty (ә-'ji-lә-tē) Property consisting of quickness,

lightness, and ease of movement; To be very nimble The ability to create and respond to change in order to

profit in a turbulent global business environment The ability to quickly reprioritize use of resources when

requirements, technology, and knowledge shift A very fast response to sudden market changes and

emerging threats by intensive customer interaction Use of evolutionary, incremental, and iterative delivery

to converge on an optimal customer solution Maximizing BUSINESS VALUE with right sized, just-

enough, and just-in-time processes and documentationHighsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.

7

What are Agile Methods?

8

People-centric way to create innovative solutions Product-centric alternative to documents/process Market-centric model to maximize business value

Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.orgRico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf

Customer Collaboration

Working Systems & Software

Individuals & Interactions

Responding to Change

valuedmore than

valuedmore than

valuedmore than

valuedmore than

Contracts

Documentation

Processes

Project Plans

Frequent comm. Close proximity Regular meetings

Multiple comm. channels Frequent feedback Relationship strength

Leadership Boundaries Empowerment

Competence Structure Manageability/Motivation

Clear objectives Small/feasible scope Acceptance criteria

Timeboxed iterations Valid operational results Regular cadence/intervals

Org. flexibility Mgt. flexibility Process flexibility

System flexibility Technology flexibility Infrastructure flexibility

Contract compliance Contract deliverables Contract change orders

Lifecycle compliance Process Maturity Level Regulatory compliance

Document deliveries Document comments Document compliance

Cost Compliance Scope Compliance Schedule Compliance

Courage

NetworkComputer

Operating SystemMiddlewareApplications

APIsGUI

How Agile Works Agile requirements implemented in slices vs. layers User needs with higher business value are done first Reduces cost & risk while increasing business success

9Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.

Agile Traditional1 2 3 Faster

Early ROI

Lower Costs

Fewer Defects

Manageable Risk

Better Performance

Smaller Attack Surface

Late

No Value

Cost Overruns

Very Poor Quality

Uncontrollable Risk

Slowest Performance

More Security Incidents Seven Wastes1. Rework2. Motion3. Waiting4. Inventory5. Transportation6. Overprocessing7. Overproduction

MINIMIZES MAXIMIZES

JIT, Just-enough architecture Early, in-process system V&V Fast continuous improvement Scalable to systems of systems Maximizes successful outcomes

Myth of perfect architecture Late big-bang integration tests Year long improvement cycles Breaks down on large projects Undermines business success

10

Capability/MMF #1

● Feature 1● Feature 2● Feature 3● Feature 4● Feature 5● Feature 6● Feature 7

Capability/MMF #2

● Feature 8● Feature 9● Feature 10● Feature 11● Feature 12● Feature 13● Feature 14

Capability/MMF #3

● Feature 15● Feature 16● Feature 17● Feature 18● Feature 19● Feature 20● Feature 21

Capability/MMF #4

● Feature 22● Feature 23● Feature 24● Feature 25● Feature 26● Feature 27● Feature 28

Capability/MMF #5

● Feature 29● Feature 30● Feature 31● Feature 32● Feature 33● Feature 34● Feature 35

Capability/MMF #6

● Feature 36● Feature 37● Feature 38● Feature 39● Feature 40● Feature 41● Feature 42

Capability/MMF #7

● Feature 43● Feature 44● Feature 45● Feature 46● Feature 47● Feature 48● Feature 49

1

2 3

4

5 6

7

8 9

10

11 12

13

14 15

16

17 18

19

20 21

Evolving “Unified/Integrated” Enterprise Data Model

“Disparate” LEGACY SYSTEM DATABASES (AND DATA MODELS)

ETL

A A

B C

D E F

G H I J K

A

B C

D E F

A

B C

D E

A

B C

D

A

B C

A

B

“Legacy” MS SQL Server Stovepipes “Inter-Departmental” Linux Blade/Oracle/Java/WebSphere Server

“Leased” DWA/HPC/Cloud Services

Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7

Release

Release

Release

Release

ETL ETL ETL ETL ETL ETL

Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.

Agile Development In-the-Large“Incremental Business Value”

(for example, assume 25 user stories per feature, 175 user stories per capability/MMF, and 1,225 user stories total)

Organize needs into capabilities, features, and stories Prioritize features, group releases, and initiate sprints Develop minimum set of features with highest value

Agile for EMBEDDED SYSTEMS 1st-generation systems used hardwired logic 2nd-generation systems used PROMS & FPGAs 3rd-generation systems use APP. SW & COTS HW

11Pries, K. H., & Quigley, J. M. (2010). Scrum project management. Boca Raton, FL: CRC Press.Pries, K. H., & Quigley, J. M. (2009). Project management of complex and embedded systems. Boca Raton, FL: Auerbach Publications.Thomke, S. (2003). Experimentation matters: Unlocking the potential of new technologies for innovation. Boston, MA: Harvard Business School Press.

● Short Lead● Least Cost● Lowest Risk● 90% Software● COTS Hardware● Early, Iterative Dev.● Continuous V&V

● Moderate Lead● Moderate Cost● Moderate Risk● 50% Hardware● COTS Components● Midpoint Testing● “Some” Early V&V

● Long Lead● Highest Cost● Highest Risk● 90% Hardware● Custom Hardware● Linear, Staged Dev.● Late Big-Bang I&T

AGILE“Software Model”- MOST FLEXIBLE -

NEO-TRADITIONAL“FPGA Model”

- MALLEABLE -

TRADITIONAL“Hardwired Model”

- LEAST FLEXIBLE -

GOAL – SHIFT FROM LATE HARDWARE TO EARLIER SOFTWARE SOLUTION

RISKEmbeddedSystemsMore HWThan SW

STOPCompeting

With HW

STARTCompeting

With SW

Iter

atio

ns, I

nteg

rati

ons,

& V

alid

atio

ns

12

Architecture—Software RadioREL #1 REL #2 REL #3 REL #4

13Kovacs, K. (2015). Comparison of nosql databases. Retrieved on January 9, 2015, from http://kkovacs.euSahai, S. (2013). Nosql database comparison chart. Retrieved on January 9, 2015, from http://www.infoivy.comDB-Engines (2014). System properties comparison of nosql databases. Retrieved on January 9, 2015, from http://db-engines.com

Rank Database Year Creator Firm Goal Model Lang I/F Focus Example User Rate KPro

2007 Steve Francia

10gen Gener-ality

Document C++ BSON Large-scale Web Apps

CRM Expedia 45% 48

2008 Avinash Lakshman

Facebook Relia-bility

Wide Column

Java CQL Fault-tolerant Data Stores

Mission Critical Data

iTunes 20% 15

2009 Salvatore Sanfilippo

Pivotal Speed Key Value C Binary Real-time Messaging

Instant Messaging

Twitter 20% 14

2007 Mike Carafella

Powerset Scale Wide Column

Java REST Petabyte-size Data Stores

Image Repository

Ebay 10% 8

2004 Shay Banon

Compass Search Document Java REST Full-text Search

Information Portals

Wiki-media 5% 7

Real-time, Distributed, Multi-tenant, Document-based, Schema-free, Persistence, Availability, etc.

8

Redis10

HBase14

Rapid-prototyping, Queries, Indexes, Replication, Availability, Load-balancing, Auto-Sharding, etc.

Distributed, Scalable, Performance, Durable, Caching, Operations, Transactions, Consistency

Real-time, Memory-cached, Performance, Persistence, Replication, Data structures, Age-off, etc.

Scalable, Performance, Data-replication, Flexible, Consistency, Auto-sharding, Metrics, etc.

16Elastic Search

MongoDB5

Cassandra

3 - $10M•Gen App•Reliable•Low Cplx

2 - $100M•Schema•Dist P2P•Med Cplx

1 - $1B•Limited•Sin PoF•High Cplx

Agile for CLOUD COMPUTING 1st-generation systems used HPCs & Hadoop 2nd-generation systems used COTS HW & P2P 3rd-generation systems use APP. SW & COTS HW

What is Agile Testing? Traditional testing is a late, manual process Agile testing is an early and automated process Goal to deliver early & often and V&V components

14Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txtCrispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.

AGILE TESTING- Early Incremental Testing -

TRADITIONAL TESTING- Late Big Bang Integration Testing -

Test Criteria Accompany StoriesAutomated Tests Written FirstUnits Coded-Tested One at TimeCode is Frequently Checked InCode Automatically RetrievedCode Automatically CompiledTests Automatically Executed Instant Feedback & Test Reports

Test Criteria Written After FactManual Tests Written Much LaterUnits Coded Late All at One TimeCode Checked In Late in ProjectCode Manually Submitted to TestCode Manually Compiled & BuiltTests Manually Executed LateLate Project Feedback & Reports

Code Automatically DeployedLate Defects Freeze Projects

BASIC—Test Driven Development Term coined by Kent Beck in 2003 Consists of writing all tests before design Ensures all components are verified and validated

15Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.

ADVANCED—Continuous Integration Term coined by Martin Fowler in 1998 Process of automated build/regression testing Evaluates impact of changes against entire system

16Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

ALL DEVELOPERS RUN PRIVATE BUILDS

DEVELOPERS COMMIT CODE TO VERSION CONTROL

INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY

100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD

A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD

FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY

REPORTS AUTOMATICALLY GENERATED & REVIEWED

Thousands of TestsContinuously Executed

No More Late BigBang Integration

User needs designed & developed one-at-a-time Changes automatically detected, built, and tested System fully tested and deployed as changes occur

17Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.

BuildIntegration

Server

VersionControlServer

BuildScripts

UsesWatches

BuildStatus

ProvidesDeveloper A

Developer B

Developer C

CommitsChanges

CommitsChanges

CommitsChanges

Builds

Database

Analysis

Testing

Reporting

Documentation

Deployment

Early, Automated, Fast,Efficient, & Repeatable

Constant ReadinessState & CM Control

Lean, Waste Free, Low WIP,No Deadlocked Test Queues

Rapidly & SuccessfullyDev. Complex Systems

Agile Testing Process“Continuous Integration”

Agile Testing Practices Agile testing consists of seven broad practices Automated build, database, inspection, tests, etc. Include reporting, documentation, deployment, etc.

18

Practice

Building

Database

Inspections

Testing

Feedback

Documentation

Deployment

Description

Frequently assembling products and services to ensure delivery readiness

Frequently generating/analyzing database schemas, queries, and forms

Frequently performing automated static analysis of product/service quality

Frequently performing automated dynamic product and service evaluation

Frequently generating automated status reports/messages for all stakeholders

Frequently performing automated technical/customer document generation

Frequently performing automated delivery of products/services to end users

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.

Agile methods are based on traditional measures Story points, velocity, and burndown basic metrics Experts use Agile EVM, test, ROI & portfolio metrics

19Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.

AGILE METRICS1. Agile CODE Metrics2. Agile PROJECT Metrics3. Agile TRACKING Metrics4. Agile TESTING Metrics5. Agile VALUE Metrics6. Agile HEALTH Metrics7. Agile PORTFOLIO Metrics

1. Agile CODE Metrics Code Size Code Complexity Object Oriented Code Coverage Code Defects Relational Design

2. Agile PROJECT Metrics Software Size Software Productivity Software Effort Software Quality Software Schedule Software Success

3. Agile TRACKING Metrics Story Points Sprint Burndown Release Burndown Velocity Feature Progress Agile Earned Value

4. Agile TESTING Metrics Test Coverage Test Automation Integration Builds Running Tested Features DevOps Automation Deployment Frequency

7. Agile PORTFOLIO Metrics Portfolio Kanban Epic Progress Portfolio Radar Release Train Radar Lean Portfolio Metrics Enterprise Scorecard

6. Agile HEALTH Metrics Teamwork Quality Collaboration Quality Agile Process Maturity Agile Adoption Rate Degree of Agility Product Flexibility

5. Agile VALUE Metrics Total Lifecycle Costs Total Lifecycle Benefits Benefit to Cost Ratio Return on Investment Net Present Value Real Options Analysis

Agile Metrics Taxonomy

20

METRIC DESCRIPTION

TEST COVERAGE Percent or degree to which software source code is tested

TEST AUTOMATION Ratio or degree to which software tests are automated

INTEGRATION BUILDS Frequency of automated software builds and integrations

RUNNING TESTED FEATURES Number of completed and tested features or user stories

DEVOPS AUTOMATION Ratio or degree to which deployments are automated

DEPLOYMENT FREQUENCY Frequency of automated software deployments or deliveries

Software test automation emerged during the 1970s Reached their height in personal computer (PC) era Most are FOSS and used by successful agile teams

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Agile Testing Metrics—Definitions

Agile Testing Metrics—Example

21Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Agile Testing Workflow

22

Traditional vs. Agile Cumulative Flow

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Wor

k (S

tory

, Poi

nt, T

ask)

or E

ffor

t (W

eek,

Day

, Hou

r)

Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)

Traditional Cumulative Flow Agile Cumulative Flow

Late big bang integration increases WIP backlog Agile testing early and often reduces WIP backlog Improves workflow and reduces WIP & lead times

Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.

Agile Testing Costs & Benefits

Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.

Most agile testing tools are “free” open source Build server costs no more than a commodity PC 10x more efficient/effective than traditional testing

23

Agile Testing Economics Traditional testing finds a defect in about 10 hours Manual code inspections find a defect in 1 hour Agile testing finds a defect every 6 minutes

24Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Agile Cost of Quality (CoQ) Agile testing is 10x better than code inspections Agile testing is 100x better than traditional testing Agile testing is done earlier “and” 1,000x more often

25Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.

Continuous Integration Fewer integrations leave in higher bug counts Frequent, early integrations eliminate most defects Goal is to have as many early integrations as possible

26Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.

Number ofIntegrations

Less Defects•More Integrations•Early IntegrationsMore Defects

•Few Integrations•Late Integrations

Agile Testing Anti-Patterns Agile teams don’t often use TDD, CI, CD & DevOps Implement independent test teams after Sprints done Sprint Waterfalling, Scrummerfalling, & Wagile result

27Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.

Incorrect• Phased Testing• Separate Teams• Delayed Testing

Correct• Integrated Testing• Integrated Teams• Continuous Testing

Scaling Agile Testing Agile testing slows down with very large systems Slow testing slows integration and increases bugs Agile testing can speed back up with more attention

28Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

MICRO ADJUSTMENTS- Focused Impact Tuning-

MACRO ADJUSTMENTS- Wide Impact Tuning-

Add More CPUs & MemoryParallelize System BuildsReplace 3rd Party Test LibrariesReduce or Remove Test TimeoutsSelect Different TestsRefactor Code & ComponentsTune Network & SoftwareTune Database & Middleware

In-Memory CompilationParallelize Test RunsPre-Install Test LibrariesRemove Process RandomnessUse Faster Code & Test Tools Incremental vs. Big Bang TestsParallelize Build & InstallTune & Optimize Build Process

Enterprise Continuous Delivery Created by Jez Humble of ThoughtWorks in 2011 Includes CM, build, testing, integration, release, etc. Goal is one-touch automation of deployment pipeline

29Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.

CoQ

• 80% MS Tst• 8/10 No Val• $24B in 90s• Rep by CD• Not Add MLK

30

Hewlett-Packard is a major user of CI, CD, & DevOps 400 engineers developed 10 million LOC in 4 years Major gains in testing, deployment, & innovation

Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.

TYPE METRIC MANUAL DEVOPS MAJOR GAINS

CYCLE TIME

IMPROVEMENTS

Build Time 40 Hours 3 Hours 13 x

No. Builds 1-2 per Day 10-15 per Day 8 x

Feedback 1 per Day 100 per Day 100 x

Regression Testing 240 Hours 24 Hours 10 x

DEVELOPMENT

COST EFFORT

DISTRIBUTION

Integration 10% 2% 5 x

Planning 20% 5% 4 x

Porting 25% 15% 2 x

Support 25% 5% 5 x

Testing 15% 5% 3 x

Innovation 5% 40% 8 x

Cont. Delivery (Hewlett-Packard)

Continuous Delivery (Assembla) Goal of continuous delivery is releases vs. build/tests Market-driven releases creates rapid business value Assembla went from 2 to 45 monthly releases w/CD

31Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.

62x FasterU.S. DoD

IT Project

3,645x FasterU.S. DoD

IT Project

Agile Scaling at Google Google early adopter of agile methods and Scrum Google also uses agile testing at enterprise scale 15,000 developers run 120 million tests per day

32Micco, J. (2013). Continuous integration at google scale. Eclipse Con, Boston, MA.Whittaker, J., Arbon, J., & Carollo, J. (2012). How google tests software. Upper Saddle River, NJ: Pearson Education.

440 billion unique users run 37 trillion searches each year Single monolithic code tree with mixed language code Submissions at head – One branch – All from source 20+ code changes/minute – 50% code change/month 5,500+ submissions/day – 120 million tests per day 80,000 builds per day – 20 million builds per year Auto code inspections – For low defect density 10X programming productivity improvement $150 million in annual labor savings (ROI as a result)

Agile Scaling at Amazon Amazon adopted agile in 1999 and Scrum in 2004 Using enterprise-scale continuous delivery by 2010 30,000+ developers deploy over 8,600 releases a day

33Atlas, A. (2009). Accidental adoption: The story of scrum at amazon.com. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 135-140.Jenkins, J. (2011). Velocity culture at amazon.com. Proceedings of the Velocity 2011 Conference, Santa Clara, California, USA.Elisha, S. (2013). Continuous deployment with amazon web services. Proceedings of the AWS Summit 2013, Sydney, New South Wales, Australia.

Software deployment every 11.6 seconds (as of 2011) 24,828 to 86,320 releases per Iteration 161,379 to 561,080 releases per Quarter 645,517 to 2,244,320 releases per Year

Automatic, split-second roll-forward & backward 75-90% reduction in release-caused outages (0.001%) Millions of times faster (than traditional methods) 4,357,241 to 15,149,160 per traditional release

Thousands of times faster (than manual agility) 161,379 to 561,080 per Scrum/SAFe release

Used agile methods long before U.S. government (1999)

Enables enterprises to be flexible but disciplined Allows enterprises to distribute project work teams Ensures distributed project teams are collaborating

34

Agile Tools“Across the Life Cycle”

Project Management

RequirementsDOORSRequisite ProSLATE

DesignRhapsodyTelelogic System ArchitectRational System Architect

CodingEclipseVisual StudioSun Studio

TestingJUnitNUnitXunitCPPUnit

GtestFitFitnesseSelenium

Quality AssuranceCheckStylePMDEMMAJdependCoberturaGcov

Configuration MgtSubversion (SVN)Concurrent Versions Sys.ClearCase

Build AutomationAntNAntMavenMake

Continuous Integ.Cruise ControlHudsonBuildBot

CollaborationWebExSkypeMeetMeWimba

WikiMediaWikiTracWikiPhpWiki

DocumentationNDocJavadocDoxygeniText

Version OneRallyScrum WorksVSTS

Agile TeamAgile EnterpriseScope ManagerStory Studio

XP Plan ItIterateXP TrackerAgilo

XP CGIXP WebXplannerIce Scrum

Project CardsTarget ProcessXtreme PlannerTeam System

CommunityEnterpriseMingleHansoft

There are literally hundreds of agile testing tools There are tools for building, testing, and deployment Integration tools monitor repositories and initiate tests

35

Agile Tools“In-Depth Test Automation”

Smart, J. (2009). Automated deployment with maven and friends: Going the whole nine yards. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.

Agile Testing Usage Statistics Agile test use is low in spite of its age, i.e., 15 years Many do not understand its utter simplicity and power Failure to use agile testing undermines project success

36Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.

Agile PracticesRetrospectives

Refactoring

Done Definition

Test Tools

Test Driven Dev.

CM Tools

Simplicity

Pair Programming

Technical Debt

Agile Testing 13%

Continuous Integrations

Weekly

Daily

2-3 TimesPer Day

Never

2-3Times

PerIteration

Common Barriers to Agile Testing Industry very slow in adopting agile testing model Cost, difficulty, and territorialism are common issues Developers must take initiative for disciplined testing

37

Technical BarriersOrganizational BarriersDevelopers don’t want to test

· Infrequently committing code· Committing broken code· Failing to immediately fix builds· Not writing automated tests· Not ensuring 100% of tests pass· Not running private builds· Resorting to traditional testing

Resistance to change· Fear of investment costs· Fear of learning new skills· Test group territorialism· Organizational policy conflicts· Overhead of maintaining CI· Complexity and scaling· Not developing a quality culture

··

Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.

Eliminates big-bang integration in the 11th hour Creates a repeatable and reliable testing process Evaluates system-wide changes throughout project

38Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.

What’s the Bottom Line?“Agile Testing Done Early & Often”

Agile TestingTraditional TestingDramatically reduces risks

· Automates manual processes· Instant verification & validation· High project visibility· Greater confidence and morale· Incremental business value· 24x7 deployability to users· Highly quality and reliability

Late defect discovery· Low quality software· Poor project visibility· Lack of deployability· Late big-bang integration· Testing is a bottleneck· Poor customer satisfaction· Outright project failure

··

Dave’s Professional Capabilities

39

SoftwareQuality

Mgt.

TechnicalProject

Mgt.

SoftwareDevelopment

Methods

OrganizationChange

SystemsEngineering

CostEstimating

GovernmentContracting

GovernmentAcquisitions

LeanKanban

Big Data,Cloud, NoSQL

WorkflowAutomation

Metrics,Models, & SPC

SixSigma

BPR, IDEF0,& DoDAF

DoD 5000,TRA, & SRA

PSP, TSP, &Code Reviews

CMMI &ISO 9001

InnovationManagement

Statistics, CFA,EFA, & SEM

ResearchMethods

EvolutionaryDesign

Valuation — Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.

Lean-Agile — Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.

STRENGTHS – Data Mining Gathering & Reporting Performance Data Strategic Planning Executive & Manage-ment Briefs Brownbags & Webinars White Papers Tiger-Teams Short-Fuse Tasking Audits & Reviews Etc.

● Action-oriented. Do first (talk about it later).● Data-mining/analysis. Collect facts (then report findings).● Simplification. Communicating complex ideas (in simple terms).● Git-r-done. Prefer short, high-priority tasks (vs. long bureaucratic projects).● Team player. Consensus-oriented collaboration (vs. top-down autocratic control).

PMP, CSEP,ACP, CSM,

& SAFE

32 YEARSIN IT

INDUSTRY

Books on ROI of SW Methods Guides to software methods for business leaders Communicates the business value of IT approaches Rosetta stones to unlocking ROI of software methods

http://davidfrico.com/agile-book.htm (Description) http://davidfrico.com/roi-book.htm (Description)

40