what do defects really cost? much more than you think

T13

Test Management

5/7/2015 1:30:00 PM

What Do Defects Really Cost?

Much More Than You Think

Presented by:

Wayne Ariola

Parasoft

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Wayne Ariola

Parasoft

Chief strategy officer Wayne Ariola leads the development and execution of Parasoft’s long-

term strategy. Wayne has contributed to the design of core Parasoft technologies and has been

awarded several patents for his inventions. He is a recognized leader on service virtualization,

API quality, quality policy governance, and application security. Wayne brings more than twenty

years of strategic consulting experience in the technology and software development

industries. Prior to joining Parasoft, he helped co-found a start-up and consulted in the strategic

change practice at PricewaterhouseCoopers.





Parasoft Proprietary and Confidential 1

2015-04-30


Wayne Ariola

Chief Strategy Officer - Parasoft


Wayne Ariola

Chief Strategy Officer - Parasoft

Session AbstractSession Abstract

Software quality has traditionally been a “time-boxed”

exercise and we can’t possibly expect that accelerating the

SDLC via DevOps will yield better results from a quality

perspective. Automating the build-test-release-deploy

process will undeniably get the application into the hands of

users faster. However, if you haven’t also re-engineered the

Parasoft Proprietary and Confidential 2Parasoft Proprietary and Confidential 2

users faster. However, if you haven’t also re-engineered the

processes by which the application is built and tested, you’re

quite likely flooding your users with more defects at a more

rapid rate.

Re-Invention…Of TimeRe-Invention…Of Time


http://www.scienceworldreport.com/articles/22225/20150210/cool-optical-lattice-

clocks-1000-times-more-precise-cesium-atomic.htm

Software Drives InnovationSoftware Drives Innovation


The Cost of Quality

associated with software has shifted dramatically

Software has shifted from process enabler to business

differentiator

SDLC Speed will be the difference between a first mover and a follower

Switching costs

associated with software are dramatically lower

Software Failures = HeadlinesSoftware Failures = Headlines

Financial Airlines/Aero Government Media Internet/Tel

Parasoft Proprietary and Confidential 5

Automotive Technology Retail

Software Failures = Headlines 2014Software Failures = Headlines 2014

The day of the announcement companies lost an average

Software failures make headline news—

eroding customer confidence, shareholder value and brand equity


-$2.3 Bn-3.75%

lost an average

of shareholder value

Escalating Cost of Failure: Public Escalating Cost of Failure: Public

-1.50%

-1.00%

-0.50%

0.00%

2012 2013 2014 2015 (est.)

Impact on Market Capitalization


-3.12%-3.37%

-3.75%

-4.18%y = -0.0036x - 0.0272-4.50%

-4.00%

-3.50%

-3.00%

-2.50%

-2.00%

Software Failures = HeadlinesSoftware Failures = Headlines

5,500,000

3,490,0004,000,000

5,000,000

6,000,000

Market does not forget— news about failure increase and

average of 167%


821,0001,080,000

451,000

1,400,000

896,000

1,550,000

650,000

3,490,000

2,240,000

0

1,000,000

2,000,000

3,000,000

4,000,000

Cloud Services Airline Bank Software Retail

1st Event 2nd Event

The Cost of Software Failure - SonyThe Cost of Software Failure - Sony

Market Value

-22% -33% -30%-19% -11% -12%

Cumulative Loss = $18B


http://www.reuters.com/article/2011/04/26/us-sony-stoldendata-idUSTRE73P6WB20110426

http://www.digitaltrends.com/gaming/sony-fined-almost-400000-for-2011-playstation-security-breach/

Market Value

Event 15 Days 30 Days

Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC

Plan Develop Build Test Release

Test as a time-boxed event is the prevailing method


Fixed Fixed

Testing is a BottleneckTesting is a Bottleneck

Business Views

Fully Automated Provisioning

Data Analytics

Organic SystemsSoftware Quality Processes


Basic Tasks

Skilled Tasks

Process Oriented

End-to-End Tasks

Data Analytics

Automation is Incremental

Eff

icie

ncy

Impact of Test in the Evolving SDLCImpact of Test in the Evolving SDLC

Plan Develop

Bu

ild

Release

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild

Bu

ild


Test

Test

Test

Test

Test

Test

Test

Test

Test

Test

Test

More than automation – “continuous” tests must assess business risk

From Automated to Continuous TestingFrom Automated to Continuous Testing


Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”

We have to ask a new question

No Longer: “Are we done testing?”


“Does the release candidate have an acceptable level of risk?”

Getting from “Automated” to “Continuous”Getting from “Automated” to “Continuous”

There is a big leap moving from “Automated” to “Continuous”

� Business expectations clearly defined

� Business risks identified

� Per application, team, release

� Defects automatically prioritized versus business drivers

� No defect left behind


� No defect left behind

� Mitigate business risks

� Clear ownership and workflow for remediation

� Collaborative risk reduction

� Collapse the remediation cycle time

� Feedback loop for defect prevention

� Pattern definition

� Defect prevention practices enhanced

Transforming the Identification of RiskTransforming the Identification of Risk

From To

Business

Objectives


From To

Causal Observations Probabilistic

Defect Documentation Most Likely Causes

Only Structured Data Structured and Unstructured

Filters Business Policies

Ad Hoc Automated

Tool Dependent Tool Independent

Next Generation TestingNext Generation Testing


Process IntelligenceProcess Intelligence

Process Intelligence aggregates “siloed” information throughout the SDLC

and identifies business risks early and automatically

� Correlate disparate data across:

� Dependent teams

� Dependent components

� Infrastructure systems


� Infrastructure systems

� Testing practices and tools

� Apply advanced statistical analysis

� From pre-build patterns

� From constructed

� With Open domain APIs

� Advantages

� Defect remediation

� Exception-based notifications

The Next Generation Software Quality ‘System’The Next Generation Software Quality ‘System’

Continuous Assessment of RiskContinuous Verification of Requirements

Development Operations

Development

(E1)Production

Test

(E2)

Integration

(E3)

Pre-Production

(E4 – En)

Release Candidate


Test Environment Management - Provisioning and Elastic Cloud Scale

Test Optimization

Environment Simulation (SV)

Auto Test Construction – Synthetic

APM – Real Time Data

Development Tests

“Shift-Left”“Shift-Left”


30

Environment Access?Environment Access?

The average number ofdependent applicationassociated with theSystem Under Test (SUT)


30BUT, Dev/QA only have “trustworthy”

access to SIXSIXSIXSIXof the applications

302928272625

242322212019

181716151413

121110987

654321

Test Environment ConfigurationTest Environment Configuration

The average percentage of time spent configuring the test environment.

30%


50%The average percentage of the test plan ableto complete once configured.

Survey Results…Survey Results…

When testing with a dependentdependentdependentdependent application only need access to a small percentageof the applications

20%


of the applications functionality

Service Virtualization delivers a

simulated dev / test environment

allowing an organization to test

Service VirtualizationService Virtualization


allowing an organization to test

anytime or anywhere

So, What are you Virtualizing?So, What are you Virtualizing?

AUT


• Mainframe• SOA / Web services• Java• .NET languages• XML• WSDL• WADL• UDDI• WSIL• SOAP• PoX (Plain XML)• REST• JSON• BPEL• Web Applications

• Mobile Interfaces• AJAX• JSP• JavaScript• HTML• CSS• WS-*• Standards• MTOM(XOP) / MIME / DIME • OAuth• TCP/IP• webMethods Broker• webMethods IS• HL7• FTP

• HTTP 1.0• HTTP 1.1• HTTPS• JMS• IBM WebSphere MQ• Sonic MQ• RMI• EJB• SMTP• Tibco Rendezvous• .NET WCF• ISO 8583• EDI• SAP • custom

Stubs Vs Service VirtualizationStubs Vs Service Virtualization

� Stubs� Inside-out approach that disassociates a test case

with a dependent systems

� Brittle

� Limited reuse

Service Virtualization provides a complete environment for developing and testing versus complex, dependent systems


� Limited reuse

� Static response

� Service Virtualization� Creates an environment in which to run “rich” test scenarios

� Virtualized assets represent real system behavior

� All virtualized-assets are reusable

� Virtualized assets are programmable and extensible

Complexity is a Barrier to InnovationComplexity is a Barrier to Innovation

Scheduled Access


3rd Party System

Evolving ComponentMainframe

Access to Dev / Test SystemsAccess to Dev / Test Systems

Configuration

Complex

ERP

Mainframe

ExternalDatabase

CloudApp


Test Access

Simple Difficult

Configuration

InternalService

InternalApplication 3rd Party

Service

ExternalApplication

InternalDatabase

App

MessageQueue

ESB

Web Server

Service Virtualization & DependenciesService Virtualization & Dependencies

Service Virtualization

Configuration

� Access� Dependent applications difficult

� Scheduling conflicts

� High access fees

� Geo-political boundaries

� 3rd party or partner applications

� Configuration

Complex


Virtual Lab

Configuration

Access

� Configuration� Complex to configure

� No control

� Limited variability

� Consumes test time

DifficultSimple

Easy

The Total Cost of System AccessThe Total Cost of System Access

Service Virtualization dramatically reduces the cost of deliveringa dev / test environment

Hardware Savings

Staged Application Instance

Cost � Only manage the functionality

you need

� Only manage data you need

� Low-Risk “disposable instance” that can be accessed at anytime

� Dramatically reduce test


Access EfficiencyStaged w/ Virtual Test Lab

System Behavior

� Dramatically reduce test environment setup and management

Service Virtualization

Service Virtualization ExamplesService Virtualization Examples

Mobile Application development and extension

Agile/Parallel development limited by system dependencies

Parasoft Virtualize is used throughout the SDLC to speed access to and reduce the costs of managing development and test environments


Capacity Constrained staged environments

Limited access to mainframes, ERPs, or 3rd party systems

Test data management for complex transactions

The Test Environment ChallengeThe Test Environment Challenge

Test environment access is outside the control of development and test leaving gaps in the process

IT OperationsParallel development

delays… Need simple,

realistic access to

dependent components…

Too much time

Scheduling

DependentApplications

Staged Assets


Too much time

waiting for access…Need

reliable test data

Need a realistic test

environment easy to

maintain

Configuration

Access Limits

3rd Party Assets Virtual Environments

Hyper Visor

App App App

The Test Environment ChallengeThe Test Environment Challenge

Test environment access is outside the control of development and test leaving gaps in the process

IT Operations

Scheduling

DependentApplications

Staged Assets


Configuration

Access Limits

3rd Party Assets Virtual Environments

Hyper Visor

App App App

Create, Manage, Provision

Value of Service VirtualizationValue of Service Virtualization

� Increase Access to “Trustworthy” Test Environments� Provide a complete test environment

� Provide access 24/7

� Flexibility provides a platform for extending test scenarios, coverage

� Reduce Operational Expenditures

Service Virtualization drives better software qualitywhile reducing CapEx and OpEx


� Reduce Operational Expenditures� Significantly reduce test environment configuration time

� Simplify test environment access

� Remove the bottleneck of test environment setup

� Reduce Capital Expenditures� Software assets can be run anywhere

� Significantly reduce bandwidth demand

� Reduce dependency on staged environments

Parasoft Service VirtualizationParasoft Service Virtualization

Questions?


Wayne AriolaChief Strategy OfficerParasoft [email protected]

Virtual Lab ManagementVirtual Lab Management

Strengths Weaknesses

� “Low” Risk

� Reduce CAPEX

� Reduce Server Sprawl

� Manage Environment

� Must Have Access

� Permissions to Control

� Configuration

Permutations


� Manage Environment

Complexity

� Uniformity for Global

Teams

� Operational Efficiency

Permutations

� Test Data

� Performance

� Hardware Capacity

Constraints

The Value/ROI Drivers – HumanThe Value/ROI Drivers – Human

Virtual Lab Service Virtualization

� Reduce OPEX � Eliminate scheduling delays

associated with staged test labs

� Eliminate ‘some’ configuration

delays

� Eliminates configuration time by

delivering more granular ‘service’

versus system

� Eliminates scheduling delays

� Understand

Environment

Complexity

� Gives a good view of inter-

relationship of applications

within geo/political control

� Gives a complete view of all

system dependencies


Complexity

� Operational

Efficiency

� Reduces demand on test lab

resources

� Reduces delays of scheduling,

setup and re-configuration

� Eliminates delays associated with

� System configuration

� Data management

� Performance configuration

� Setup and tear down

The Value/ROI Drivers – PhysicalThe Value/ROI Drivers – Physical


� Reduce CAPEX

and server

sprawl

� Versus physical test lab will

significantly reduce CAPEX

� Can reduce incremental CAPEX

due to physical test capacity

constraints

� Can reduce demand on

“virtualized” infrastructure

resources

� Eliminate � Can eliminate physical test lab

BUT be careful what you want to

� Eliminates the need for

incremental increases in physical


� Eliminate

physical test

labs

(Careful!)

BUT be careful what you want to

virtualize due to compliance and

data privacy concerns

incremental increases in physical

test labs or virtual lab capacity

The Value/ROI Drivers – Project/ProcessThe Value/ROI Drivers – Project/Process


� Time to Market

(Careful!)

� Increases time to market by

providing access to “production-

like” environments within

geo/political control

� Increases time to market by

providing more complete access

� Provides a net new environment

earlier in the process

� Access anytime

or anywhere

� Good access to completed and

built applications within control

� Access to both complete and

incomplete components

� Manage and � Good view of applications � Much broader view of total


� Manage and

understand

Environment

� Good view of applications

managed within resource pool

� Much broader view of total

system interdependencies

� Understanding of 3rd party

impacts

� Defect

Reproduction

� Captures great information for

applications under control

� Extends information by including

components outside of control

� Better Testing � Increased code coverage

� Increase test efficiency

� Increased test coverage

� Increased # and quality of tests

� Increase test efficiency

� Increased test coverage

� Increased # and quality of tests

What’s the View from Your Seat?What’s the View from Your Seat?

Strategy

Risk Profile

Governance

Business

Portfolio

Application


Actions

Project

Process

Risk Profile Application

Release

Iteration

Code

From Automated to ContinuousFrom Automated to Continuous

Continuous testing accelerates the SDLC bymanaging quality expectations and actionable tasks

Requirements

Defined

Policy Management

Development

Defect Prevention

Development Testing Integration Testing System Testing

Defect Remediation Tasks


Development

Static Analysis

Unit/Component

Peer Review

Automated Tests

API/Service Tests

Smoke Test

Security Tests

Automated Tests

Functional Tests

Scenario Tests

Performance Tests

CI

Build

Go

Release Path

No GoBusiness

Decision

Service Virtualization – Test Environment Access

what do defects really cost? much more than you think

Software

average software failures

cost of quality

headlinessoftware failures

api quality

quality perspective

term strategy

quality policy governance

average of shareholder