CS4723Software

Validation and Quality Assurance

Lecture 15Advanced Topics

Test Plans and Management

Test Plan & Management

Something you are going to think about when you are a project manager instead of a tester / developer

About handle the testing progress of a whole software project

About quality assurance at highest level

Approach: risk based testing

Presentation Outline Introduction - The Challenge Why? Risk Analysis and Testing

Risk Based Test Approach What? Test Metrics

Risk Based Test Metrics

Risk Analysis in Risk Based Testing Risk Analysis in Practice

Plan and Manage - Metrics How? Quality and Progress Monitoring

Warning! Organization and Process...

The Challenges

Time Constraints Resource Constraints Quality Requirements Risk Factors:

New technology Lack of knowledge Lack of experience

Take Control!

Presentation Outline Introduction - The Challenge Risk Analysis and Testing What?

Risk Based Test Approach Test Metrics

Risk Based Test Metrics

Risk Analysis in Risk Based Testing Theory and Practice

Plan and Manage Quality and Progress Monitoring

Warning! Organization and Process...

Risk Analysis and Testing

Risk Identfication

Risk Strategy

Risk Assessment

Risk Mitigation

Risk Reporting

Risk Prediction

Testing, Inspection etc.

Test Plan

Metrics: Cost and Probability

Test Item Tree

Test Metrics

Risk Based Testing - Approach

Risk Identification / Test Items Identify the items to be tested

Functions, classes, features, transactions, … Risk Strategy / Test Plan:

Setup approaches to generating test cases Automatic / Manual Original / Regression Load / Security / Usability / ,,,

Risk Assessment / Metrics Assess risk of each test items

Risk Based Testing - Approach

Risk Mitigation / Testing Generate and execute the concrete test cases

Risk Reporting Measure how much have we mitigated the risk

Test coverage Found and fixed bugs

Risk Prediction Based on the risk reports, predict risks of test items for new

round of testing

Risk Based Testing - Metrics Identify Areas with “High Risk Exposure”

Probability and Cost All functions/modules should be tested to a

“minimum level” “Extra Testing” in areas with high risk exposure Establish Test Plan and Schedule

Monitor Quality Number of Faults per function and time

Monitor Progress Number of hours in test and fix -> ETC

The Formula

Re(f) - Risk Exposure of function f P(f) - Probability of a fault in function f C(f) - Cost related to a fault in function f

Risk Based Testing - Theory

P(f)*C(f)Re(f)

Simple Test Metrics

Test Planning Number of test cases per function Number of hours testing per function

Progress Tracking Number of tests planned, executed and

completed Number of faults per function Number of hours used for test and fix Estimated to Complete

Simple Test Metrics

Probability of faults - Indicators New functionality Size Complexity Quality of previous phases and documents

Cost of Faults

Presentation Outline Introduction - The Challenge Risk Analysis and Testing

Risk Based Test Approach Test Metrics

Risk Based Test Metrics

Risk Analysis in Risk Based Testing How? Theory and Practice

Plan and Manage Quality and Progress Monitoring

Warning! Organization and Process...

Risk Based Testing - Example

Other Probability Factors might include: Function Points, Frequency of Use etc.

Ranking the functions based on Risk ExposureThe Probability of a FaultThe Cost of a Fault

Example: 2

C(s)C(c)P(f)*Re(f)

Cost Probability Risk

Func. C( s ) C(c) Avr.New Func.

Design Quality Size

Com-plexity

Weighted Averag

Exp. func.

5 5 1 3 fInterest Calc. 3 3 3 2 3 3 3 37 111

Close Account 1 3 2 2 2 2 3 31 62

Cust. Profitab. 2 1 1,5 3 3 2 3 41 61,5

Consequence

Pro

bab

ilit

y

LowLow

High

High

1

2

34

Risk Based Testing - Reporting

510

440

10 1

TECHNICAL

INTERFACE

RISK

439

370

11 2 2

369

302

5

Low Medium High

BUSINESS RISK

Low Medium High

Low

M

ediu

m

Hig

h

Consequence

Pro

bab

ilit

y

Prior to test execution: identify critical transactions

1

Test Execution identifies

“bad” transactions

2

Extra Testing: - Additional testing by product specialist- Automated regression testing

3

Risk Based Testing - Practice

Presentation Outline Introduction - The Challenge Risk Analysis and Testing

Risk Based Test Approach Test Metrics

Risk Based Test Metrics

Risk Analysis in Risk Based Testing Theory and Practice

Plan and Manage How? Quality and Progress Monitoring

Warning! Organization and Process...

Two Strategies

On-line Risk Analysis Usually used for first time testing Cold start with little data Minimal testing and a small list of critical features

provide by customers Batch Risk Analysis

Assume a large amount of data at the beginning Different metrics

On-line Test Cases Completed

Date

Nu

mb

er o

f T

est

Cas

es

Planned

Executed

QAed

Planning and Progress Tracking

Progress Indicators - “To be vs. Actual”

“To be fixed” vs. “Actually fixed”

To be retested Act. retested

Rejected

To Be Restested, Actually Retested and Rejected

Nu

mb

er o

f F

ault

s“To be Retested” vs. “Actually Retested”

Actually fixed

To Be Fixed and Actually Fixed

Nu

mb

er o

f F

ault

s

To Be Fixed

Progress Indicators - Hours Used

BatchNumber of hours

for finding one fault and for fixing one

OnlineNumber of hours for finding one fault and for fixing one

Hours per Fault for Test and Fix

Date

Ho

urs

pe

r F

ault

Test

Fix

Test

Fix

Hours per Fault for Test and Fix

Ho

urs

pe

r F

ault

“Estimated to Complete” ETC for system test based on:

Number of hours testing per fault found Number of hours fixing per fault Number of

faults found per function

Number of fixes being rejected

Number of remaining tests (functions to be tested)

Calculated ETC and Actual Hours

Date

Ho

urs

ET

C

Estimated to Complete at Time t

Actual to Complete at Time t

Benefits of Risk Based Testing

Improved Quality? all critical functions tested

Reduced Time and Money in Testing effort not wasted on non critical or low risk

functions Improved customer confidence

due to customer involvement and good reporting and progress tracking

Presentation Outline Introduction - The Challenge Risk Analysis and Testing

Risk Based Test Approach Test Metrics

Risk Based Test Metrics

Risk Analysis in Risk Based Testing How? Theory and Practice

Plan and Manage Quality and Progress Monitoring

Warning! Organization and Process...

WARNING! Require a focused Organization…

Strong Test Management Testers to “drive” the Programmers Responsibility

Application based

groups

v222

Test log

Test procedure

TEST EXECUTION

Change

request

2

Test design

Test data

Function based

groups

DYNAMIC

TEST PLANNING

STATIC

TEST PLANNING

Test plan

Test item tree

Test item spec.

Test case spec.

Test

summary

report

LOG KEEPING

Event log

Error /

change log

Objectives,

policy and

strategy for the

test project

specific.

…and a well defined Process DocumentationPlanning and Execution ProceduresControl ProceduresPlanning and Progress Tracking

LD PD

Test Exec

PTDsRaised

Fix

CR

Fix Procedure

Problem Mngmnt.Procedure

Test Exec.Procedure

Case BuildProcedure

Case QualityStandards

Change Mngmnt.Procedure

Re-test

Good/Bad

TestCompleted

RegressionTest

Good

Bad

ProAte

QC / QABasic Test Data

Test Case

Risk Identification

Risk Assessment

Risk Mitigation

Risk Reporting

Risk Prediction

Test Process Work Flow

Summary

Risk Based Test Approach Focused Testing

Reduced Resources Improved Quality

Metrics are fundamental Process and Organization must support the

new strategy Metrics must support the organization and

process