Empowering the Test Case

Introduction

A pass rate is an abstraction of the actual result of a set of test cases

This abstraction is done to facilitate for stakeholders

A 50 % pass rate says that half the test cases passed – but what value does this

information actually hold?

I most cases, not much!

We have gone to far in our abstraction and the information now holds almost no real

value to our stakeholders, even if they might still think so

Test Case Attributes

How can the result of a set of test cases be more valuable?

By assigning standardized attributes to test cases, the value of the test result will be

much higher

The test case will provide more information that can be aggregated when generating the

test report to the stakeholders

What information depends on the attributes – this presentation outlines one set of

attributes

Test Case Attributes Overview

Test

Case

Software

Quality

Attributes

Priority

Priority

Each test case is assigned a priority based on several factors

What severity will a defect found by the test case most likely have?

How important is the functionality to stakeholders?

With what frequency will the test case most likely find defects?

There can be many factors that set the priority

The priority can be a scale between 1 and a number which is set based on the

granularity that is deemed to be needed by stakeholders

Software Quality Attributes

By listing what sort of software quality attribute the test case covers the test result will

reveal much more detail

You can use any set of software quality attributes

IEC 9126-1 is one example of software quality attributes

Functionality

Maintainability

Efficiency

Usability

Reliability

Portability

Test Suite Overview

Performance

Priority 1

Test Case A

Functionality

Priority 3

Test Case B

Efficiency

Priority 1

Test Case C

Performance

Priority 2

Test Case D

Functionality

Priority 2

Test Case E

Functionality

Priority 1

Test Case F

Test Results without Test Case Attributes

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Pass Rate

Test Results

Blocked

Failed

Passed

Test Results with Software Quality Attributes

0%10%20%30%40%50%60%70%80%90%

100%

Blocked

Failed

Passed

Test Results with Priority

0%10%20%30%40%50%60%70%80%90%

100%

Priority1

Priority2

Priority3

Blocked

Failed

Passed

Test Case Attributes Added Value

Imagine that the three above charts show the same test results

Initial indications show that 85 % are passed – Quite good results

The second chart however shows that almost all of the failed test cases are located in

the reliability area – This can for example indicate that we have severe stability

problems

The third chart shows that almost all of our failed test cases have high priority – also an

increased risk

Even if the initial indications were positive, analyzing the test results with test case

attributes further reveals more information and additional risks

Conclusion

By adding test case attributes to the test cases, the test result can hold much more

value to the stakeholder

It can also facilitate for the tester during scope selection

The cost is very small, and the potential gain can be very large if stakeholders base

critical decisions on the test results

What test case attributes to add depends on stakeholder needs – what information can

be valuable to the stakeholders?

Reference

IEC 9126

http://en.wikipedia.org/wiki/ISO/IEC_9126