1

Value-Based Software Testing

using Requirements and Risks

Mag. Rudolf Ramlerrudolf.ramler@scch.at+43 7236 3343 872http://www.scch.at

R. Ramler 2

Software Competence Center Hagenberg

Software Research Company

About 65 employees

We focus on the areasSoftware Architectures and Technologies

Automated Software Engineering

Industrial Data Warehousing

Knowledge-based Vision Systems

Industrial Data Mining

Process and Quality Engineering

SCCH closes the gap between Software Research and Software Product Development 2

2

R. Ramler 3

Overview

The Vital Few and the Useful Many

The Value of Priorities

Requirements-based Test Management

Risk-based Test Management

Test Case Selection

R. Ramler 4

The Vital Fewand the Useful Many

Pareto, Vilfredo (1848-1923)Italian economist and sociologist, known for his theory on mass and elite interaction and his application of mathematics to economic analysis.

20% of the population controls 80% of the

wealth and income …

3

R. Ramler 5

The Vital Fewand the Useful Many

20% of his peapods yielded 80% of the peas that were harvested

20% of criminals account for 80% of crime

20% of motorists account for 80% of accidents

20% of streets account for 80% of the traffic

20% of your carpet gets 80% of the wear

…

R. Ramler 6

SLAC Computer NetworkingWAN Monitoring, Dec. 1998

20% of pings80% of delay

4

R. Ramler 7

Static Analysisof Large Software Systems

20% files 74% LOC (relevant lines)

20% classes 82% outbound calls, 50% classes no calls

R. Ramler 8

The Vital Fewand the Useful Many

20% of his peapods yielded 80% of the peas that were harvested

20% of criminals account for 80% of crime

20% of motorists account for 80% of accidents

20% of streets account for 80% of the traffic

20% of your carpet gets 80% of the wear …20% of product flaws account for 80% of problems

5

R. Ramler 9

Boehm & BasiliSoftware Defect Reduction Top 10

"About 80 percent of the defects come from 20 percent of the modules, and about

half the modules are defect free. …

About 90 percent of the downtime comes from, at most, 10 percent of the defects."

Barry Boehm, Victor R. Basili: Software Defect Reduction Top 10 List.

IEEE Computer, January 2001

R. Ramler 10

The Vital Fewand the Useful Many

20% of his peapods yielded 80% of the peas that were harvested

20% of criminals account for 80% of crime

20% of motorists account for 80% of accidents

20% of streets account for 80% of the traffic

20% of your carpet gets 80% of the wear …20% of product flaws account for 80% of problems

20% of clients usually account for 80% of an organization’s profits

6

R. Ramler 11

Overview

The Vital Few and the Useful Many

The Value of Priorities

Requirements-based Test Management

Risk-based Test Management

Test Case Selection

R. Ramler 12

Value of PrioritiesExample

7

R. Ramler 13

Value of PrioritiesThe Vital Few and the Useful Many

Is every test cases, defect, requirement equallyimportant?

Pareto principle: 20% of clients account for 80% of the profits

In the example: 15 different customer billing types Type #1 accounts for about 50% of the transactions

Type #1 to #3 together for about 80%

0

10

20

30

40

50

60

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

R. Ramler 14

Value of Priorities20% Customer Types 80% Business Value

8

R. Ramler 15

Value of PrioritiesROI of ATG vs. Pareto

R. Ramler 16

Value of PrioritiesROI of ATG vs. Pareto

Maximum ROI with only 40% of tests!

ATG Testing

Pareto Testing

9

R. Ramler 17

Setting PrioritiesApproaches

1. Requirements-based testingWhat value do the requirements have?

2. Risk-based testingWhat puts the value realization in danger?

3. Test case selection techniquesWhat are your best test cases?

Requirements

Risks

TCS

R. Ramler 18

Overview

The Vital Few and the Useful Many

The Value of Priorities

Requirements-based Test Management

Risk-based Test Management

Test Case Selection

Requirements

Risks

TCS

10

R. Ramler 19

Requirements-based TestingApproach

Trace requirements to test cases and vice versaDerive test cases from requirementsDerive priorities from requirementsAssess requirements coverageNotify testing on requirements changesReport test results against requirements

Tool support neededValue-based tracing

Avoid bureaucracyTrace only important requirements

R1

R2

R3

TC1

TC2

TC3

TC4

?

?Requirements Test cases

R. Ramler 20

11

R. Ramler 21

Overview

The Vital Few and the Useful Many

The Value of Priorities

Requirements-based Test Management

Risk-based Test Management

Test Case Selection

Requirements

Risks

TCS

R. Ramler 22

Risk-based TestingApproach

[Bach 1999]

12

R. Ramler 23

Risk-based TestingHow to identify your risks?

Risk catalogs – common and domain specific risksBug taxonomies – common bugs and bug patternsQuality criteria categories

Starting point, e.g., customized characteristics of ISO/IEC Std. 9126-1:2001

Assemble a list of quality aspects important for the product or specific in the project– What is important for the customer and end user?

– What is typical for the market segment, in similar products?

– Experience, support database, expert advice, ...

ISO 9126

♦ Functionality

♦ Reliability

♦ Usability

♦ Efficiency

♦ Maintainability

♦ Portability

R. Ramler 24

Quality x Feature Matrix

Risk-based TestingCase Study: Web Testing

View eventsView eventsView eventsView events

Resource Utilizat ion

Ti me Behavior

Attracti veness

Operability

Learnability

Understandability

Recoverability

Fault Tolerance

Maturi y

Security

Compliance

Interoperability

Accuracy

Suitability

View eventsView eventsView events3

Feature

Security aspect V

iew

eve

nts

Vie

w d

etai

ls

Add

eve

nt

Edi

t eve

nt

Atta

ch

docu

men

t

Set

pro

perti

es

Del

ete

even

t

General identity B B C C C C C

Message content authenticity B B C C C C C

Message content origin C C B B B A B

Integrity C C A A A A

Secrecy and privacy B B C B C C C

Accountability B B B B B

view details

attach document

i nvite attendees

upload new document

select exisitng document

set security properties

add event

<<extend>>

<<i nclude>>

edit event

<<extend>>

<<extend>>

delete event

subscribe to event

Registered User

login

Userview events

<<extend>>

<<extend>>

<<extend>>

Quality Aspects

Functionality

Reliability

Usability

Efficiency

13

R. Ramler 25

Risk-based TestingCase Study: Web Testing

Feature

Quality Level Vi

ew e

vent

s

View

det

ails

Add

even

t

Edit

even

t

Atta

ch

docu

men

t

Set p

rope

rties

Del

ete

even

t

Basic

Operation

Detail

Leve

l of D

etai

l in

Test

ing

Risk Profile

R. Ramler 26

Risk-based TestingCase Study: Web Testing

In Kappel G. et al.: Web Engineering, Wiley 2006

From "Risk-orientation" to Risk Management at project level

14

R. Ramler 27

Overview

The Vital Few and the Useful Many

The Value of Priorities

Requirements-based Test Management

Risk-based Test Management

Test Case Selection

Requirements

Risks

TCS

R. Ramler 28

Test Case SelectionApproach

Various approaches based on different heuristicsChange – favor modified functions, methods, or covered items (provided by version management)

Coverage – statement, additional statement, branch, functions, requirements

Cost – test cost estimates (e.g., execution time)

History – time since a test was last run or previously detected faults

Ongoing reseach …

15

Test Case SelectionResearch Studies

Random (un-prioritized)

Coverage (measured in terms of total number of functions)

Change (measured in terms of additional modified functions covered)

Optimal (upper bound on prioritization effectiveness)

Rothermel G. and S. Elbaum

Putting Your Best Tests ForwardIEEE Software Aug./Sept. 2003

R. Ramler 30

SummaryWhat You Can Do

Value-based Quality Management

Risk- and Requirements-based Testing

Mining Repositories for Defect Prediction

Software Quality Assessment

Requirements

Risks

TCS