a process for risk-based test strategy development and its industrial evaluation

Rudolf Ramler1 and Michael Felderer2

1 Software Competence Center Hagenberg, Hagenberg, Austria2 Institute of Computer Science, University of Innsbruck, Austria

December 04, 2015 PROFES 2015, Bolzano

A Process for Risk-Based Test Strategy Development and Its Industrial Evaluation

Motivation: ISO/IEC/IEEE 29119

Slide 2Risk‐Based Test Strategy Development and Industrial Evaluation

http://softwaretestingstandard.org/

Risk Concept in Software Testing


Probability (P) Impact (I)

Risk Value

Risk Item

Probability Criteria Impact Criteria

Risk LevelTest Planning

Test Design

Test Execution

Test EvaluationRisk‐Based Test Strategy

Risk‐Based Test Strategy Development and Industrial Evaluation

Defect Classification

Identification of Impact Factors

1. Definition of Risk Items

2. Probability Estimation

3. Impact Estimation

4. Computation of Risk Values

6. Definition of Test Strategy

7. Refinement of Test Strategy

Defect Management

Requirements Management

5. Determination of Risk Levels

Quality Management

Identification of QA/Testing Techniques Proces Step

Preparation Step

Related Process

Legend

Process for Risk‐BasedTest Strategy Development

Definition of Risk Items (1), Probability (2) and Impact Estimation (3)


ProbabilityIm

pact

Derive impact from weighted impact factors in fishbone

diagram!

Derive probability from defect distribution!

Computation of Risk Values (4)


Component RankA 5 5 21 2 1 105 1 0 5 6B 29 2 9 50 5 0 1.450 1 4 5 0 1C 25 2 5 10 1 0 250 2 5 0 4D 18 1 8 46 4 6 828 8 2 8 2E 14 1 4 8 8 112 1 1 2 5F 13 1 3 50 5 0 650 6 5 0 3

Estimated Probability Estimated Impact Computed Risk (= P * I)

Determination of Risk Levels (5)


Definition and Refinement of Test Strategy (6), (7)


Components A B C D E FI II III IV Risk level I IV II III I III

Unit testing (100% branch coverage) X XCode reviews X X X X X X XManual testing of use cases (base flow) X XManual testing of use cases (base + alternative flows) X X X X XExploratory testing X X X X XAutomated smoke/regression tests X X X X XBeta test phase at selected customers X X X X X X X

Risk levelTesting techniques

• Case study with five software development companies• Research questions

(RQ1) Is the outcome of the process useful? (RQ2) Is conducting the process easy?(RQ3) Is the outcome of the process representative?

• Iterative data collection and analysis in joint research transfer project, in each iteration(1) Joint workshop to discuss and prepare steps of the process(2) Application of steps to create practical results for company(3) Questionnaire to collect feedback from each company(4) Semi‐structured interviews with key members of companies

Evaluation


Case A Case B Case C Case D Case EDomain ERP software Access

systemsDocument management

Training and recruitment

Payment systems

Core business

Software product and service

Hardware incl. embedded software

Software product and custom development

Service including software platform and operation

Solution development incl. software and hardware

Employees 15 40 10 40 15Software Releases

2 to 4 releases per year; service releases on demand

Adjusted to hardware product cycles (years)

New releases every one and four weeks

On demand Delivery as custom projects

Overview of Case Companies


1.

2.

3.

4.5.

6.

7.

A B C D E Avg

high

low

Perceived Usefulness (RQ1)


Usefulness of ... A B C D E Avg sdev1. Definition of Risk Items 2 1 1 1 2 1.4 0.49

2. Probability Estimation 2 2 1 1 1 1.4 0.49

3. Impact Estimation 1 2 1 1 1 1.2 0.40

4. Computation of Risk Values 1 5 1 2 1 2.0 1.55

5. Determination of Risk Levels 1 1 1 2 1 1.2 0.40

6. Definition of Test Strategy 1 1 1 3 1 1.4 0.80

7. Refinement of Test Strategy 1 2 3 2 2 2.0 0.63

1.

2.

3.

4.5.

6.

7.

A B C D E Avg

high

low

Perceived Ease of Use (RQ2)


Ease of Use of ... A B C D E Avg sdev1. Definition of Risk Items 1 2 3 5 2 2.6 1.36






7. Refinement of Test Strategy 1 1 3 2 4 2.2 1.17

1.

2.

3.

4.5.

6.

7.

A B C D E Avg

high

low

Perceived Representativeness (RQ3)


Representativeness of ... A B C D E Avg sdev1. Definition of Risk Items 1 4 1 1 1 1.6 1.20






7. Refinement of Test Strategy 4 2 2 2 2.5 0.87

• Process for risk‐based test strategy development• Evaluation in research transfer project with five companies

• Overall positive result for usefulness of the process• Overall ease of use perceived as moderate• Most effort caused by Impact Estimation, Definition of Test Strategy and Definition of Risk Items

• Overall representativeness is high with one notable exception• Future work

• Refinement of process steps Impact Estimation and Definition of Test Strategy

• Additional evaluations especially long‐turn studies

Conclusions


Risk‐Based Test Strategy Development and Industrial Evaluation

Track on Systematic Literature Reviews and Mapping Studies in SE