kpis, cmmi … und dann? - itq | it quality group...

KPIs, CMMI … und dann?

Gerd Höfner, Debanjan Ray, Rakesh K. Singh (Siemens Information System Ltd., Bangalore, India)

Dr. Dirk Hamann, Prof. Dr. Peter Liggesmeyer,Michael Ochs, Michael Ochs, Michael Ochs, Michael Ochs, Adam Trendowicz(Fraunhofer IESE)

SQM Congress 2006SQM Congress 2006SQM Congress 2006SQM Congress 2006

Folie 2Copyright © 2006, Fraunhofer IESE & Siemens Information Systems Ltd.

SQM Congress 2006

Siemens Information Systems Ltd.

Overview• Short introduction of partners• Short intro to KPIs, CMMI L4 & 5

• SISL: Quality Initiatives History -> Way to CMMI-L5• Example for process improvement on L5 with quantitative evidence

(effort estimation accuracy, effort estimation procedure, strategic decision for size input to effort estimation)

• Going beyond CMMI-L5 • The COMPAS Project• Target COMPAS Models and their

• Conclusions


SQM Congress 2006






• Conclusions


SQM Congress 2006



• Global HQ: Mumbai, India• Global Operations: 32 Countries• Founded in: 1992• Employees: 4200• FY05 Revenue: USD 171 Mio

#15/Top 20 IndiaSoftware & Service Exporters (Criterion: Annual Turnover)Goal: Reach Top10 next year!

SISL Management Board

Business Groups

Business Solutions

Engineering & Industrial Application

Communication Select Vertical Market

SAP Consulting

ITS

MCS

SCM

Training

SEC

HC PD/PS

SAC

Network

Business Appln

Manufacturing

FS

Utility

SISL Management Board

Business Groups

Business Solutions

Engineering & Industrial Application

Communication Select Vertical Market

SAP Consulting

ITS

MCS

SCM

Training

SEC

HC PD/PS

SAC

Network

Business Appln

Manufacturing

FS

Utility


SQM Congress 2006


Fraunhofer IESE Kaiserslautern, Germany

• Fraunhofer Institute for Experimental Software Engineering

• Founded in 1996 (since 2000 permanent Insitute)

• Mission: Applied Research & Technology Transfer with quantitative Evidence

– Quantitative approach GQM & QIP (NASA)

– Research projects (EU, German)– industrial projects DE, EU, INT

• Leading European Organisation in appliedresearch on Software Engineering (rank 4 world wide)

• more than 100 scientific employees


SQM Congress 2006





•Going beyond CMMI-L5 • The COMPAS Project• Target COMPAS Models and their

• Conclusions


SQM Congress 2006


What is a KPI?KPI = Key Performance Indicator„A Key Performance Indicator (KPI) is a economic term. The term KPI denotes metrics using which the progressor fulfilment level of vital goals or critical success factors within an irganisation can be measured and monitored.“

Typical KPIs in Software Business• Process level

– Ability to remove defects (early) -> Defect Containment Effectiveness– Cost of defect removal -> Cost of Quality– Process Capability & Maturity

• Project level– Effort accuracy -> e.g. Effort Relative Error– Schedule adherence

• Product level– Reliability (MTBF, MTTR, Availability)– Product size (ekLoC, FPs)– Defect Density


SQM Congress 2006


CMMI – Short Overview (Staged Model)

Process unpredictable, poorly controlled and reactive

Process characterized for projects and is often reactive

Process characterized for the organization and is proactive

Process measuredand controlled

Focus on processimprovement

4: QuantitativelyManaged

3: Defined

1: Initial

2: Managed

5: Optimizing


SQM Congress 2006


MATURITY LEVEL 4: QUANTITATIVELY MANAGEDThe maturity level 4 process areas are as follows:

Organizational Process Performance• SG 1 Establish Performance Baselines and Models

Baselines and models that characterize the expected process performance of the organization's set of standard processes are established and maintained.

• GG 3 Institutionalize a Defined ProcessThe process is institutionalized as a defined process

Quantitative Project Management• SG 1 Quantitatively Manage the Project

The project is quantitatively managed using quality and process-performance objectives.

• SG 2 Statistically Manage Subprocess PerformanceThe performance of selected subprocesses within the project's defined process is statistically managed.

• GG 3 Institutionalize a Defined ProcessThe process is institutionalized as a defined process.

SG: Specifc goalsGG: Generic goals


SQM Congress 2006


MATURITY LEVEL 5: OPTIMIZINGThe maturity level 5 process areas are as follows:Organizational Innovation and Deployment• SG 1 Select Improvements:

Process and technology improvements that contribute to meeting quality and process-performance objectives are selected.

• SG 2 Deploy Improvements:Measurable improvements to the organization's processes and technologies are continually and systematically deployed.

• GG 3 Institutionalize a Defined Process:The process is institutionalized as a defined process.

Causal Analysis and Resolution• SG 1 Determine Causes of Defects:

Root causes of defects and other problems are systematically determined.• SG 2 Address Causes of Defects

Root causes of defects and other problems are systematically addressed to prevent their future occurrence.

• GG 3 Institutionalize a Defined ProcessThe process is institutionalized as a defined process.


SQM Congress 2006




(effort estimation accuracy, effort estimation procedure, strategic decision forsize input to effort estimation)


• Conclusions


SQM Congress 2006


Engineering Practices


ISO 9001ISO 9001 CMM L5CMM L5 CMMI L5CMMI L5 Six-SigmaSix-Sigma

PM Practices

PM Practices

Domain Knowledge

Domain Knowledge

Business ProcessesBusiness

Processes

HR (P-CMM)

HR (P-CMM)

ISO 9001:2000

(F&A)BS 7799

ISO 9001:2000

(F&A)BS 7799

SAS 70ISO 13485SAS 70

ISO 13485

Key Initiative

Focus Area

AdditionalInitiatives

94 - 99 99 - 01 01 - 04 04 - 06

SISL´s Quality Journey

Comparison: Effort Estimation Procedure


SQM Congress 2006




ISO 9001ISO 9001 CMM L5CMM L5 CMMI L5CMMI L5 Six-SigmaSix-Sigma

PM Practices

PM Practices

Domain Knowledge

Domain Knowledge

Business ProcessesBusiness

Processes

HR (P-CMM)

HR (P-CMM)

ISO 9001:2000

(F&A)BS 7799

ISO 9001:2000

(F&A)BS 7799

SAS 70ISO 13485SAS 70

ISO 13485

Key Initiative

Focus Area

AdditionalInitiatives

94 - 99 99 - 01 01 - 04 04 - 06

SISL´s Quality Journey

Comparison: Effort Estimation Procedure

SISL Good Practices� Quality Policy and Quality System� Quality Management as Early Warning System� People Involvement and Integrated Process Improvement� Balanced Scorecard and Metrics Program� ExpertWEB and Technology Innovation� Six-Sigma Programme and Process Optimization� Process Automation through QMS Portal


SQM Congress 2006


Effect of continuous process optimizationExample: Estimation procedure for project effort

Effort estimation• Basic input: Size estimate (ekLOC)• Further inputs

– external risks (e.g. delay of LC documents, late change requests, global development sites)

– internal risks (e.g. team fluctuation)

• Output: Effort (estimate in person hours)

• Evaluation of Effort estimates– Effort Accuracy / Effort Relative Error (Deviation from plan)– per phase and/or per project


SQM Congress 2006


Actual Size vs. Actual Total Effort – pre/at L5Weak relationship between actual size and actual total effort.

Regression Model:R2 = 17,4%p-Value 12%


SQM Congress 2006


Actual Size vs. Actual Total Effort – going L5 Stable, significant relationship between actual size and actual total effort.

Regression Model:R2 = 64,6% (x 3,7)p-Value 0,05%


SQM Congress 2006


Actual Size vs. Actual Eng. Effort – pre/at L5Weak relationship between actual size and actual engineering effort.

Regression Model:R2 = 21,8%p-Value 8%


SQM Congress 2006


Actual Size vs. Actual Eng. Effort – going L5 Stable, significant relationship between actual size and actual engineering effort.

Regression Model:R2 = 72,5% (x 3,4)p-Value 0,01%


SQM Congress 2006


Engineering Effort Estimation RE/MRE – pre/at L5

MeanRE = -5.05%MedianRE = -1.64%

MeanMRE = 13.33%MedianMRE = 10.04%

Engineering Effort Estimation Relative Error

-60.00%

-50.00%

-40.00%

-30.00%

-20.00%

-10.00%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

P01

P02

P03

P04

P08

P09

P10

P11

P12

P13

P14

P15

P16

P17

P18

P19

P20

P21

P22

P23

P24

P25

P26

P27

P28

P29

P30

REMREActual

ActualEstimateRE

=

×−= %100 Interpretation RERE < 0: Effort was underestimatedRE = 0: 100% hit!RE > 0: Effort was overestimated


SQM Congress 2006


Engineering Effort Estimation RE/MRE – going L5Only projects finalized in FY 2005

MeanRE = -8.06%MedianRE = -7.68%

MeanMRE = 8.43%MedianMRE = 7.68%

Engineering Effort Estimation Relative Error

-60,00%

-50,00%

-40,00%

-30,00%

-20,00%

-10,00%

0,00%

10,00%

20,00%

30,00%

40,00%

50,00%

P01

P02

P03

P04

P05

P07

P08

P09

P10

P11

P12

P14

P16

P17

P18

P20

P22

P24

P25MeanMRE: 36% lower

MedianMRE: 24% lower

Compare before values:MeanMRE = 13.33%MedianMRE = 10.04%


SQM Congress 2006


KPI Relations for Strategic Decisions (1/3)

Example: Make a strategic decision on size measurement approachWhich one is the right one in the scope of effort estimation? FP or LoC?• Three KPIs

– Size (actual)

– Effort (actual)

– Size-Effort-R-Square (Power of Size to explain variation in effort)

• Support of strategic decision

– Conduct benchmark analysis on actual data on Size(LoC)/Effort from company data vs. Size(FP)/Effort from benchmark database

– Selection of appropriate projects for the analyses• Similar/identical domain

• Similar/identical system class

• Comparable project sizes


SQM Congress 2006


KPI Relations for Strategic Decisions (2/3)Scatterplot (Spreadsheet in ISBSG_SISL.stw 4v*42c)

NWE = 32.9174+17.2383*x; 0.95 Pred.Int.

P85P86

P87

P88

P89

P90

P91P92P93 P94

P95 P96

P97

P98

P99

P100

P101P102

P103P104

P105

P107

P109P110P111P112 P113 P114P115 P116P117

P118P119

P120

P121

P122P123

P125P126

-100 0 100 200 300 400 500 600

UFP

-2000

0

2000

4000

6000

8000

10000

12000

UFP:NWE: r2 = 0.5360; r = 0.7321, p = 0.0000001; y = 32.9174 + 17.2383*xStable, significant relationship between size (FP) and effort.Sample from: ISBSG Database; X-Org. data.

Regression Model:R2 = 53,6%p-Value 0,00%


SQM Congress 2006


KPI Relations for Strategic Decisions (3/3)

Data Comparison

ISBSG (X-Org)Size-Effort-R-Square: 53,6%

SISL pre/at Level-5: Size-Effort-R-Square: 21,8%

SISL going Level-5: Size-Effort-R-Square: 72,5%

FP appears to be a meaningful FP appears to be a meaningful FP appears to be a meaningful FP appears to be a meaningful alternative to alternative to alternative to alternative to eLoCeLoCeLoCeLoC for Size for Size for Size for Size measurement and could measurement and could measurement and could measurement and could be a basis for more reliable be a basis for more reliable be a basis for more reliable be a basis for more reliable Effort estimationEffort estimationEffort estimationEffort estimation

eLoCeLoCeLoCeLoC performs (in numbers) performs (in numbers) performs (in numbers) performs (in numbers) better than FP. Considering better than FP. Considering better than FP. Considering better than FP. Considering the Xthe Xthe Xthe X----Org context, both Org context, both Org context, both Org context, both results may be compared in results may be compared in results may be compared in results may be compared in certain boundaries. certain boundaries. certain boundaries. certain boundaries. Expectation: Expectation: Expectation: Expectation: eLoCeLoCeLoCeLoC and FP will performand FP will performand FP will performand FP will performequally well as basis for Effort equally well as basis for Effort equally well as basis for Effort equally well as basis for Effort estimation. Really change to FP?estimation. Really change to FP?estimation. Really change to FP?estimation. Really change to FP?


SQM Congress 2006






• Conclusions


SQM Congress 2006


Going beyond L5 – The COMPAS Project

Situation

– Today’s SE processes are non-quantitative processes enriched with measures (easily measurable properties, e.g. productivity)

– Empirical research shows: uniform processes are not suitable, e.g. faults are not equally distributed over software units

– Domain plays an important role, e.g. safety-critical piece of SW in automotive vs. non safety-critical piece of SW in IS

– Establishing quantitatively controlled processes requires to understand underlying “rules” on a quantitative level

current situationcurrent situationcurrent situationcurrent situation

act &measure

*COMPAS = CoCoCoCooperation on MMMMeasurement-basedquantified PPPProcesses for AAAActivities in SSSSoftware Engineering


SQM Congress 2006


Going beyond L5 – The COMPAS ProjectProject Goal

– Identification and understanding of quantitative relationships of processes and products

– Exploitation of such relationships as rules for quantitatively controlling SE processes and projects

– E.g. models for predicting specific properties of SE products (along the process)

– E.g. models for predicting specific issues of SE processes

– Use of the results of the models to establish a quantitative control cycle on the process

– E.g. number of defects detected in a design (“is”);predict lower bound of overall number of defects in the design; compare “is” to lower bound and decide on how to proceed


act &measure

decide


SQM Congress 2006


Going beyond L5 – The COMPAS ProjectProject Goal

– Identification and understanding of quantitative relationships of processes and products

– Exploitation of such relationships as rules for quantitatively controlling SE processes and projects

– E.g. models for predicting specific properties of SE products (along the process)

– E.g. models for predicting specific issues of SE processes

– Use of the results of the models to establish a quantitative control cycle on the process

– E.g. number of defects detected in a design (“is”);predict lower bound of overall number of defects in the design; compare “is” to lower bound and decide on how to proceed


act &measure

decide

after project situation (planned)after project situation (planned)after project situation (planned)after project situation (planned)

analyze & predict

decide


SQM Congress 2006


Small Example for COMPAS Models Standard procedure for review

– Act: Conduct review

– Measure: Collect data on detected defects

Example for quantitative control by a prediction model

– Two reviewers conduct a requirements review (100 pages RS)

– Usual quant. pass-criterion: 2-3 defects per 10 pages

– Review Findings:

• Reviewer.1: 15; Reviewer.2: 17 Defects

• Overlap: 5 Defects; Total: 27 different Defects

• Reuslt: 2.7 Defects/10 Seiten -> Pass

– Background: Early Defect Detection saves cost & time

Situation w/quant. controlSituation w/quant. controlSituation w/quant. controlSituation w/quant. controland prediction modeland prediction modeland prediction modeland prediction model

analyze & predict

decide

Usual SituationUsual SituationUsual SituationUsual Situation

act &measure


SQM Congress 2006


Small Example for COMPAS Models Standard procedure for review

– Act: Conduct review

– Measure: Collect data on detected defects

Example for quantitative control by a prediction model

– Two reviewers conduct a requirements review (100 pages RS)

– Usual quant. pass-criterion: 2-3 defects per 10 pages

– Review Findings:

• Reviewer.1: 15; Reviewer.2: 17 Defects

• Overlap: 5 Defects; Total: 27 different Defects

• Reuslt: 2.7 Defects/10 Seiten -> Pass

– Background: Early Defect Detection saves cost & time

Situation w/quant. controlSituation w/quant. controlSituation w/quant. controlSituation w/quant. controland prediction modeland prediction modeland prediction modeland prediction model

analyze & predict

decide

Usual SituationUsual SituationUsual SituationUsual Situation

act &measure

Simple prediction model for defect content results in: Lower bound of defect content is approx. 51 DefectsResult: Probably more than 5.1 Defects/10 pages5.1 Defects/10 pages5.1 Defects/10 pages5.1 Defects/10 pagesPass? Rework?

5 inspectors

RE

-0,8

-0,6

-0,4

-0,2

0,0

0,2

0,4

0,6

0,8

1,0

1,2

1,4

1,6

1,8

M0(MLE) MH(JE) MH(CH) MT(MLE) MT(CH) MTH(CH)

Diff. to realnumber ofdefects(est’d-real)


SQM Congress 2006


SISL Models in a Nutshell (selected ones)

• Defect Containment after Review – for controlling early QS processes – as input for predicting defects in tests

• Defect Containment/Density after Test – for controlling test processes– as input for defect density

• Corrective Size Prediction– for making Size prediction (ekLOC) more precise– as input for cost prediction

• Cost/Effort Prediction– for bringing cost accuracy towards RE < 10%– for controlling and revising spent effort– predict and control effort required to finish project under size, quality aspects

(early warning system)


SQM Congress 2006


Processes, KPIs, quantitative Models

Proc A Proc B Proc C Proc D

PredictionModel

spanningseveral processesData

Data

Measure Measure

Predicted KPI X (several predictions with increasing accuracy)

Measure &Control

PredictionModel

Control &Measure

near process

Predicted KPI Y

X

Y


SQM Congress 2006






• Conclusions


SQM Congress 2006


Conclusions• Quantitative Prediction Models spanning processes can be used to gain high

benefits

• COMPAS models will enable for quantitative insight into and across subprocesses

– for stringent and proactive quantitative quality & process control

– for understanding quantitative relationships along several subprocesses

– as a subprocess spanning early warning system

• Enable for early as possible prediction (with decreasing uncertainty) of e.g. end product defect density as early as review phases

• Enable for re-estimation of project effort based on external triggers also accounting for quality issues

kpis, cmmi … und dann? - itq | it quality group...

Documents