using tmmi to measure, improve, deliver higher quality ......the goal-question-metric (gqm) approach...

Phone: +1-610-644-2856

Measure. Optimize. Deliver.

softwarevalue.com

Using TMMi to Measure, Improve,

Deliver Higher Quality Software

©2016 DCG Software Value

Topics for Discussion

• What is TMMi?

• The 5 levels of TMMi

• World class performance

• What should you measure?

• The GQM Paradigm

• Popular Metrics

• The missing measure

• A measurement framework

1


What is the TMMi?

2

What is the TMMi standard reference model?

• A staged assessment model

• Compliant with SPICE - ISO 15504

• Can be used to support ITIL and others

• Lower levels can be used in isolation

• Aligns with CMMI; follows similar assessment and ratings process as CMMI

Why use the TMMi?

• Substantial efforts made to

improve the quality of

products

• Despite quality improvement

approaches, the software

industry remains far from

effective and efficient

• Testing accounts for between

30-50% of total project costs

• Encourages defect

prevention over detection


Purpose of the TMMi

• Provide a standard reference model for all industries

• Identify testing strengths and best practices

• Improve integrated test effectiveness and efficiency

• Identify issues and risks

• Identify a program of test process improvements

• Provide public governance, measurement and accreditation

3


A Pathway to Improved Performance

4

Independent model

Integrated best practices derived from over 14 quality and test models

Underpinned by Risk Management and Monitoring & Control

Addresses all test levels

Focused on moving organisations from defect detection to defect prevention

Fit for purpose – appropriate process in the appropriate places

Is the de facto International Standard to measure test maturity


5 Steps to World Class Testing Performance

1. Know your testing performance goals

– Understand what is important to the organization

2. Assess current testing performance against your goals

– You need to know where you are before building a map to where you want to be

3. Develop a plan and close the gap

– Consider a framework/proven methods

– Identify repeatable processes

4. Continuously improve your process

– Determine what is working, what isn’t, and what could be working even better

5. Measure, measure, measure

– Decisions based on fact are always better than assumptions or guessing

5


What Are the Right Things to Measure?

6

Manual

•Test Case Productivity

•Test Execution Summary

•Defect Acceptance

•Defect Rejection

•Bad Fix Defect

•Test Execution Productivity

•Test Efficiency

•Defect Severity Index

Performance

•Performance Scripting Productivity

•Performance Execution Summary

•Performance Execution Data -Client Side

•Performance Execution Data -Server Side

•Performance Test Efficiency

•Performance Severity Index

Automation

•Automation Scripting Productivity

•Automation Test Execution Productivity

•Automation Coverage

•Cost Comparison

Common Metrics

•Effort Variance

•Schedule Variance

•Scope Change

Software Testing Metrics, Lokesh Gulechha, Cognizant, 2014

Software Testing Metrics, P.M. Venhatech Babu, Slideshare

http://www.slideshare.net/pmvenkateshbabu/testing-metrics


Identifying the Right Measures

The Goal-Question-Metric (GQM) approach is a method for developing

and maintaining a goal-driven and purposeful set of metrics.

Benefits of the GQM:

• Links to business/technical goals and needs

• Supports measurement frameworks

• Helps to focus on targeted improvement initiatives

7


The GQM Paradigm

8


The GQM Components

9


Additional Activities

10


A GQM Example

11


Some Popular Metrics

12


What is the Missing Measure?

• Mr. Smith is upset. He is the VP of the Store Systems Division.

• He has just seen the first quarter dashboard of results and his #1

priority project, Store Inventory, has the greatest number of defects.

• Here is what was reported to Mr. Smith:

• You are the development manager. How might you respond to Mr.

Smith? Do we have all the information we need to properly evaluate

these outcomes?

13

Project Delivery Cost

(000's)

Quality

Defects

PO Special On Time $500 12

Vendor Mods Late $760 18

Pricing Adj. Early $80 5

Store Inventory On Time $990 22


Tracking Software Quality

• Size (value) can serve as a normalizing metric.

• A cost-per-unit-of-work (rate) can now be calculated

• “Defect Density” for Mr. Smith’s project is, in fact, the lowest of all of his

projects

14

Project Delivery Cost

(000's)

Quality

Defects

Size

Value

Rate Density

PO Special On Time $500 12 250

$

2,000.00 0.048

Vendor Mods Late $760 18 765

$

993.46 0.024

Pricing Adj. Early $80 5 100

$

800.00 0.050

Store Inventory On Time $990 22 1498

$

660.88 0.015

* Defect Density is calculated as defects / size


Establish a Measurement Framework

15

Project X Project Y Project Z

. .

Process Management

Performance

Measures

Enterprise

Database

Project

Data

Process

MeasuresImprove

Control

Define

Measure

Exec

ute

Process

Measurement

Repository

Service Level

DashboardEngage

Analysis

Collection

Project Defect Status

0

100

200

300

400

500

600

700

800

900

1000

Jan'08

Feb'08

Mar

'08

Apr'0

8

May

'08

Jun'08

Jul'0

8

Aug'08

Sep'08

Oct'08

# o

f D

efe

cts

Total Defects Discovered Total Closed Defects

Requirements Growth and Stability

-50

0

50

100

150

200

Jan'08

Feb'08

Mar

'08

Apr

'08

May

'08

Jun'08

Jul'0

8

Aug

'08

Sep

'08

Oct'08

Nov

'08

Dec

'08

# o

f Requir

em

ents

Added Changed Deleted Total Reqs

Project Resource and Effort Status

0

200

400

600

800

1,000

1,200

1,400

1,600

Jan'08

Feb'08

Mar

'08

Apr'0

8

May

'08

Jun'08

Jul'0

8

Aug'08

Sep'08

Oct'08

Nov

'08

Dec

'08

Pro

ject

Resourc

es/H

ours

Cum Planned Effort Allocated Cum Actual Effort Spent

"Earned Value" Baseline Total Hours

Milestone Baseline Plan Actual

%

Var

Checkpoint A – Charter & Kickoff 1/10/2008 1/10/2008 1/10/2008 0%

Requirements Complete 1/28/2008 1/28/2008 1/28/2008 0%

Vendor Selection Complete 2/4/2008 2/4/2008 2/15/2008 7%

PMP/Schedule Complete 2/12/2008 2/12/2008 2/28/2008 11%

Checkpoint B– Planning & Reqs 2/28/2008 3/15/2008 11%

Design Complete 3/15/2008 4/15/2008 20%

Development Complete 4/15/2008 4/30/2008 10%

Checkpoint C– Midpoint 4/30/2008 5/15/2008 10%

Testing Complete 4/30/2008 5/15/2008 10%

Training Complete 5/10/2008 5/30/2008 13%

Go Live 5/30/2008 6/15/2008 11%

Lessons Learned/Cust Sat Survey Complete 6/1/2008 6/30/2008 19%

Checkpoint D – Deploy & Close 6/1/2008 6/30/2008 19%

Baseline

Performance

Outsource

Governance


Billing Process – Key Quality Measures

16

Billing

Key Performance

Indicators IT

Project

Delivery

Processes

IT Metrics

Program

Project Delivery

Requirements Volatility

Defect Density

Defect Detection

Effectiveness

Defect Removal Efficiency

IT Metrics

Program

Operations

Average daily Bill Input

Average Daily Bill Backlog

Average age of IT Backlog

• % of Accurate Bills

• Delayed Bills %

• Delayed Bills Avg.

Backlog #

• Delayed Bill Avg.

Backlog $$ ($M)

• % of Returned Mail

• Customer

Satisfaction Index

• Cost of Billing per

Customer

IT

Operations

Processes

IT

Measured by

Key Metrics

IT

Measured by

Key Metrics

Bil

lin

g P

rocess

IT Processes

Directly Impact

the Business

Billing

Process

Measured by

KPI’s IT performance

Improvements are

Correlated to improve

Business KPI’s


Contact Us

Email: [email protected]

Phone: 610-644-2856

http://www.softwarevalue.com

17

@DCGSWValue

/DCGSoftwareValue

/company/DCG-Software-Valuesoftwarevalue.com

Measure. Optimize. Deliver.

http://www.url.com/

using tmmi to measure, improve, deliver higher quality ......the goal-question-metric (gqm) approach...

Documents