test process maturity measurement and related measurements

TEST PROCESS MATURITY AND RELATED MEASUREMENTS

M C Nagaraj

[email protected]

in.linkedin.com/in/mcnagaraj

mailto:[email protected]

Copyright 2011. STAG Software Pvt Ltd. All rights reserved.

Contents

Test process from different perspective

Test Maturity Model (TMMSM )

TMM goals and symptoms

Metrics system architecture

Goal focused measurements

TMM SM is a registered service mark of Illinois Institute of Technology, Chicago

2


Test Process looked from different perspective

Methodology

Technology and tools

People

3



Methodology What are the key testing methods?

When do we use them?

How do we implement them?

How do we prevent defect migration and improve defect-finding capabilities at all levels?

How can we adapt standard terminologies and test standards?

How do we create meaningful test plans and other critical test deliverables?

How do we identify and prioritize potential software test process improvements?

4



Technology and tools• How can we use risk analysis techniques to decide on goals for testing?

• How can we design adequate test cases?

• How can we decide on what types of test required to find what types of bugs?

• What measurements to collect and analyze to control test process?

• How can we select test tools and vendors?

• How to implement an effective tools program?

• What are the most significant opportunities for automating test process?

• How to get better ROI for investment made on tools?

5



People• Do we understand the attitude required in people involved in testing?

• Do we know what training required for people involved in testing?

• Do people in the organization believe that the quality of test process determines the success of overall test process?

• Do we understand the development team also has role in testing?

• Do people in the organization feel that test engineers pursue faults not the people?

• Do we understand the business value of effective testing?

• Does management team support continuous test process improvement?

6

• Is there a way through which we can prove test assets created are indeed complete and effective?

• Is there a defect detection technology one can use at each stage of SDLC to uncover defects effectively?

• How can we assess accurately where we stand with respect to quality goals?

• Can we guarantee clean software to our customers?

• Is it possible to assess how much to retest when code change?

• Which, where, when and how much to automate in test engineering activities?

Have a look at HBT (Hypothesis Based Testing) at www.stagsoftware.com

Effective test process!

Copyright 2011. STAG Software Pvt Ltd. All rights reserved. 7


Internal structure of TMM

Testing capability

Levels

Maturity Goals

Maturity sub-goals

Activities/tasks/resp.

Critical views

Manager Dev/tester User/client

Implementation

and organization

adaptation

Indicate Contain

Supported by

Achieved by

Organized by

Address

TMM levels

• Test process optimization

• Quality control

• Application of process data for defect prevention

Level 5: Optimization/Defect Prevention and Quality

Control

• Software quality evaluation

• Establish a test measurement program

• Establish an organization wide review program

Level 4:

Management and Measurement

• Control and monitor the testing process

• Integrate testing into the software life cycle

• Establish a technical training program

• Establish a software test organization

Level 3:

Integration

• Institutionalize basic testing techniques & methods

• Initiate a test planning process

• Develop testing and debugging goals

Level 2:

Phase Definition

Level 1:

Initial

Copyright 2011. STAG Software Pvt Ltd. All rights reserved. #


Metrics system architecture

Process-1 Process-2 Process-n Customer

Measurements

Questions

GOALS

Metrics

Basili’s GQM Paradigm is goal centered framework


TMM Level 1

• Testing is a chaotic process

• Not distinguished from debugging

• Documented set of specification not available

• Tests are developed in ad-hoc way after coding is completed

• The objective of testing is to show that software works

11


Measurements at L1

As there is no specific goals at this maturity level we can see whether we have answers for these questions

1. How do you know the size of software tested?

2. How do you categorize projects as big, medium and small projects?

3. Do you know how many defects were found in the project?

4. How much effort spent for activity coming under heading “testing”?

Size:Size of code in KLOCNumber of requirements or featuresNumber of test cases developedNumber of test cases executed

Defects:High, medium, low severity defects countDefects/KLOC

Cost:Total effort spent for testing

12


TMM Level 2

TMM Level 2 goals Characteristics

2.1 Develop Testing and

Debugging Goals and Policies

2.2 Initiate a Test Planning

Process

2.3 Institutionalize basic

testing techniques and

methods

There is a clear separation

between debugging and testing

phase

It is a planned activity in project

plan

Plan starts after coding is

complete

Basic testing techniques in place

Testing is multi-leveled

13

• How much time planned for all levels of testing?

• How much time spent in testing activities?

• How much time spent in debugging and fixing different types of bugs?

• What percentage of code coverage targeted and achieved?

• Did we uncover different types of defects at right stage in the SDLC?

Possible questions to check on goals status



Measurements at L2

Time/effort related measurement:

Time/effort spent is test planning

Time/effort spent in unit, integration, system, regression testing

Total time/effort spent in testing activities

(Granularity in above measurements also possible like time/effort spent in test design for unit / integration/ system tests)

Number of planned test cases, unplanned test cases

Planned/actual degree of statement coverage

Defects:

Number of defects in each phase (SDLC Phases)

Number of defects found in each level of testing (UT, IT, ST, UAT)

Number of each type of defects found

Time taken to fix and re-test each defects type

15


TMM Level 3


3.1 Establish a Test

Organization

3.2 Establish a technical

training program

3.3 Integrate testing into the

software life cycle

3.4 Control and monitor the

testing process

There is an established test

organization

Testing is integrated to SDLC

Test plan is developed, tracked

and controlled (Integrated with

project plan)

Test engineers drive test process

improvement

Users/clients attend milestone

meeting

User/clients support in

developing usability test plans

16

• How many people are in testing team at different levels?

• How many hrs of training the team go through for competency development for different levels?

• How much time spent in all levels of testing against plan?

• How many projects using tools invested effectively?

• Do you know what percentage of defects escaping UT/IT (internal) and ST (External)?

• Are we under control for planned budgets for testing in all projects?

• Is team productivity assumed matching with actual observed?




Measurements at L3

• Coverage related measurementRequirement, statement, branch coverage

Types of test planned and completed

• Productivity related measurementNumber of Test cases written / Per unit time

Test cases executed /Per unit time

• Training related measurementNumber of training hrs attended / year (for all test professionals)

• ROI on tools initiativesTotal cost saved by automation/ Total tool program cost

• Defect escapes at different levels of testingUnit Test escape = (Total UT bugs found in ST/ Total ST bugs)

External Test escape = (Total bugs found in UAT / Total bugs found in all levels

• Number of Users/Clients formal interactions

18


TMM Level 4


4.1 Establish an organization

wide review program

4.2 Establish a test

measurement program

4.3 Software quality

evaluation

Review program is effective

Reviews are planned activity in

project plan

Measurements collection and

analysis process effective

Quality attributes of a product

are well-defined and measured

19

• Is review activities conducted formally?

• How many are trained on inspection process?

• Is formal inspection effective?

• Hope measurements to be collected in all projects are agreed upfront and collected?

• Are measurements collected are analyzed?

• Is there an organization critical metrics defined with baseline value?

• Do we get clear visibility on quality at each review?




Measurements at L4

• Number of inspection leaders available

• Number of people trained on inspection

• Size of the item inspected

• Time spent on inspection activities

• Number of defects found per hour of review time

• Effort spent on different types of test

• Quality levels/quality index reached at each milestone review against plan

• Measurement collections against plan

• # of corrective action if required based on analysis of measurements collected

21


TMM Level 5


5.1 Defect prevention

5.2 Quality control

5.3 Test process optimization

Data from all projects are

collectively analyzed

Critical defects types are

analyzed thoroughly

Quality control concepts are

adopted

Right tools are inserted

progressively in the organization

22

• Is organization critical measurements are coming from all projects?

• Do we understand various types of defects we uncover and where we need to act for prevention?

• Do we have clear control limits defined for critical metrics in the organization for key processes?

• Are you happy with ROI on tools invested?

• Is senior management support for spending effort process improvements evident?




Measurements at L5

• Time/Effort spent in defect causal analysis

• Number of actions suggested

• Effort/cost for implementing action plans

• Costs of statistical testing

• Effort/costs of training SEPG team in process control

• Effort/costs on quantitative process analysis

• Number of process changes

• Number of new tools introduced in the organization

24


References

1. Practical Software Testing by Ilene Burnstein

2. Software Metrics by C. R. Pandian

25


Thank you!

Web: www.stagsoftware.com

Blog: www.stagsoftware.com/blog

Twitter: @stagsoft

http://blog.stagsoftware.com/