1

The Role of Quality Assurers In Software Assurers In Software

Engineering’s Fourth WaveDr. Bill Curtis

SVP & Chief ScientistCAST Software

© CAST 2008

Rethinking Quality

The degree to which a product meets its specified requirementsQuality

Problem⎯Customers struggle to state functional requirements. They do not understand non-functional requirements.

“…a failure to satisfy a non-functional requirement can be critical, even

t t hi f ti l i t

2© CAST 2008

catastrophic…non-functional requirements are sometimes difficult to verify. We cannot

write a test case to verify a system’s reliability…The ability to associate code to

non-functional properties can be a powerful weapon in a software engineer’s arsenal.”

2

The 4th Wave in Software EngineeringWhat: Architecture, Quality characteristics, ReuseWhen: 2005Why: Ensure software is constructed to standards

that meet the lifetime demands placed on itProduct

4

What: Design methods, CASE toolsWhen: 1980-19902

What: CMM, ITIL, PMBOK, AgileWhen: 1990-2005Why: Provide a more disciplined environment for

professional work incorporating best practicesProcess

3

3© CAST 2008

What: 3rd & 4th generation languages, structured programmingWhen: 1965-1980Why: Give developers greater power for expressing their

programsLanguages

1

Why: Give developers better tools and aids for constructing software systemsMethods

Expanding Technical Diversity in Apps

ASP/JSP/VB/.NET

User Interface Tier

WebServices

Middle-ware

Enterprise Applications

Application Logic TierJava, C++, …

Frameworks Struts MVC, Spring

Legacy Applications

CICS Monitor (Cobol) Tuxedo Monitor (C)

CICS Connector

Data Management Tier

4© CAST 2008

Preventing application level defects requires analysis of all the interactions between components of heterogeneous technologies

DatabasesFiles

COBOLBatch

Shell Scripts

Database

Data Management TierEJB – Hibernate

3

Godot’s Gotta Be in There Somewhere

Product Catalogue

? hosts

Resource pools?Connection pools?

Error handling?Ti t ?

RetailWebsite? hosts

?K threads

Credit Card Application

? hosts? threads

? threads

Order EntryApplication

? hosts? threads

Timeouts?

5© CAST 2008

Express Service Application

? hosts? threads

Shipping Application

? hosts? threads

Ever have an application hang

waiting for a response that will never come?

Even Worse, Dispersed Development

6© CAST 2008

4

What Is the Role for Quality Assurers?

Old answer: “I test and manage releases”

“I assure the process”“I assure the process”

New answer: “I ensure dependable & secure IT services to the business”

7© CAST 2008

“I provide data to quantify risk, the business value of quality, and IT decisions”

Ensure Dependability and Security

National Research CouncilSoftware for Dependable Systems

f“As higher levels of assurance are demanded…testing cannot deliver the level of confidence required at

a reasonable cost.”

“The cost of preventing all failures will usually be prohibitively

expensive, so a dependable system

8© CAST 2008

“The correctness of the code is rarely the weakest link.”

will not offer uniform levels of confidence across all functions.”

Jackson, D. (2009). Communications of the ACM, 52 (4)

5

Application quality measures how well the individual components

Application Quality

Application Quality vs. Code Quality

e t e d dua co po e tswork together to make up the

overall business system

Code quality is the measure of individual components for

li ith t d d

Code Quality

9© CAST 2008

Good code quality ≠ Good application quality

compliance with standards and best practices in the

context of a specific language

Failure to Provide Limits

TIO

N

User Interface Tier Logic Tier Data Tier

AP

PLI

CA

T

User enters No pagination Eager fetch−all Large table

10© CAST 2008

Big problems are often the result several interacting weaknesses in the code, none of which caused the problem by itself

User enters large range

of data

No pagination or cache

mechanism

Eager fetch all records from

dependent tables are returned

Large table yields huge data volume

6

Bypassing Architectural Elements

TIO

NUser Interface Tier Logic Tier Data Tier

AP

PLI

CA

T

11© CAST 2008

Preventing application level defects requires analysis of all the interactions between components of heterogeneous technologies

Passing Test Is Not the Same As Smooth Operations

GUI LOGIC DATA

Queries developed in a loopFunctions used in WHERE clause

AP

PLI

CA

TIO

N

DATA

F

12© CAST 2008

Queries do not use indexes Table size grew by 10X in a month

Coding weaknesses that passed test are exposed as the size of the data or frequency of use grows

7

Stability Anti-PatternsScaling effects

SLA Inversion

Attacks of self denial

Blocked

Usersleads to

exacerbatescounters

Decoupling middleware

reducesimpact

counters

mitigates

and Patterns

Blocked threads

Chain reaction Integration

points

Slow CascadingUnbalanced

found near

leads to

leads to

leads to

mutualaggravation

Bulkheads

counters

Steady state

results from violating

avoids

Test harnessDetects

problems

counters

13© CAST 2008

Slow responses

Cascading failures

Unbounded result sets

Unbalanced capacities

Michael Nygard (2007). Release It!

leads to

leads to

Handshaking

Fail fast

counters

Timeouts

Circuit breaker

counters

can avoid works with

counterscounters

prevents

Presenting Dependability Cases

Provide direct evidence that a system satisfies its dependability requirementsObjective

Dependabilitygoal

Dependabilityclaims

Dependabilityproperties

stated as about

argues provides

14© CAST 2008

Dependabilitycase

Dependabilityevidence

comprises• Auditable• Complete

• Sound

• Tests• Proofs

• Analyses

8

The Business Value of Application Quality

TRANSFERABILITYallows new teams to quickly

begin working with an application

Application Health Factors

Improve software readability

Reduce vendor lock-in

Reduce cost of ownership

Maximize standards compliance

Tactical Objectives

Minimize business risks

Business Value

CHANGEABILITYmakes an application easier and

quicker to modify

ROBUSTNESSimproves application stability & reduces injecting new defects

Reduce learning curves

Ease team handoffs

Reduce application rework

Maximize application availabilityMinimize liquidated damages

Minimize degraded service

Reduce injected defects

Reduce application mistakes

Maximize business agility

Minimize IT costs

15© CAST 2008

PERFORMANCEReduces degraded response

times and increases scalability

SECURITYaffects an application’s ability to prevent unauthorized intrusions

Reduce modification effortAccelerate new function delivery

Maximize application scalability

Maximize speed of response

Maximize information retrieval

Maximize information protection

Maximize customer confidence

Minimize unwanted breaches

Optimize work productivity

Maximize customer loyalty

Anti-patterns and Defects

0.015

0.02025Different Applications in a Bank

rns/

BFP D

efec erns

/BFP D

efec

15

20

25

30

0.40

0.45Application Versions in Telecom

0.000

0.005

0.010

15

20

App A App B App C

Anti-

patte

r cts/BFPAnti-patterns

Defects

0.070.08

10

12

FP

Same Application in a Bank

Anti-patternsD f t 12

14

0 025

0.03

BFP A

Application Versions in System Integrator

Anti-

patte

cts/BFP

0

5

10

15

R4 R5 R6 R70.30

0.35

Release

Anti-patternsDefects

16© CAST 2008

00.010.020.030.040.050.06

0

2

4

6

8

5.3.07 5.7.0.1 5.11.0.7

Defects/B

FP

Anti-

patte

rns/

BF

App C Versions

Defects

0

2

4

6

8

10

0

0.005

0.01

0.015

0.02

0.025

3.2 3.3 3.4 3.6App Versions

Actua

l Defects/B

Anti‐patterns/BFPAnti-patterns

Defects

9

Allianz Austria’s Maintainability Results

Business Need• Reduce maintenance costs• 700,000 LOC insurance mgt. system• 10 million customers and processes

SolutionStatically analyze application quality and stabilize maintainability scoreProvide reports to help development teams remediate problems faster

12LOC

10 million customers and processes • 8 million claims a year

Business Benefit

17© CAST 2008

0

2

4

6

8

10

Def

ects

/ K

L2002 2003 2004 2005 2006

56% reduction in defects in 4 years

• Maintainability stabilized despite 40% increase in code over 4 years

• Reduced delivery time by 60%

• Reduced costs 20% over 3 years

• 56% reduction in defects in 4 years

Deutsche Telecom⎯Managing SuppliersT-Systems Active Billing & Multimedia Solutions

• 120 billion call records and 1 billion invoices per year• SAP, Siebel, all front end apps that power www.tcom.de

Internal quality analysis run one or two times per quarterly release• Penalties in contracts based on internal quality metrics• Aggregation of metrics into C-level management dashboards

Project #nProject #3

Project #2Project #1

150+ app in 4 silos Dashboard

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1.1 2.0 2.5

18© CAST 2008

Parser Agent

T-Systems Multimedia Solutions GmbH

Team #1

Team #2

Team #3

Team #4

10

Consortium for IT Software Quality

CISQIT organizations, Outsourcers, Agencies, Experts

IT & AD Executives

Technical experts

19© CAST 2008

Define industry issuesDrive standards adoptionBuild appraiser program

Create quality standardDeveloper certification

Integrate with standards

What Does This Mean for QA Professionals

1. New opportunities as QA role upscales

2. Develop application analysis skills

3. Manage IT risk on behalf of business

4. Drive data collection and analysis

20© CAST 2008

The closer to the business, the stabler the job!