Critical KPI for an Efficient Test OrganizationSimplicity adds value

Purpose

• The purpose of this presentation is to present a number of KPI that should be used to drive an efficient test organization

• If test and development are integrated in the same organization, then this setup is not applicable, even though it could inspire to do some similar measurements in that organization

What is a KPI?

• “Measures that help decision makers define and measure progress toward business goals. KPI metrics translate complex measures into a simple indicator that allows decision makers to assess the current situation and act quickly.” [2]

Start with Why, not What or How!

• Remember to start by asking the question:“Why do I need to do measurements?”

• Then continue with:“What do I need to measure to mitigate those reasons?”

• And finally:“How do I measure that?”

• Don’t start measuring something just because you can

• And don’t ask for measurements without thinking about the goal of those measurements first

Measuring the Right Thing

• “In business, organizations should strive to create metrics that encourage behaviors that benefit the enterprise as a whole. These metrics and their reporting should provide a guiding light toward achieving the “3 Rs” of business—everyone doing the right things, and doing them right, at the right time. “ [3]

Occam’s Razor

• By definition, all assumptions introduce possibilities for error; if an assumption does not improve the accuracy of a measurement, its only effect is to increase the probability that the overall measurement is wrong. [4]

• “… most systems work best if they are kept simple rather than made complex; therefore simplicity should be a key goal in design and unnecessary complexity should be avoided.” [5]

Simplicity not Complexity

• Because assumptions decrease KPI accuracy it is often better to create a simple KPI that correlates well enough with the objective of the measurement than to try and create the perfect KPI

• Complex KPI can also create the illusion of validity because many of the factors in the KPI are highly correlated with each other and a number of highly correlated inputs should not increase confidence much more than only one of the inputs [10]

Qualitative & Quantitative KPI

• “Qualitative properties are properties that are observed and can generally not be measured with a numerical result. They are contrasted to quantitative properties which have numerical characteristics.” [1]

Critical KPI for an Efficient Test Organization

Overhead Costs

Environment Utilization

Operational Costs

Issue Leakage

Stakeholder Satisfaction

Fixed Issues

Employee Satisfaction

Innovation

Leadership

Cost OrganizationValue

Cost: Operational Costs

• Quantitative KPI

• Frequency: Monthly

• For a commercial enterprise, operating costs fall into two broad categories:– fixed costs, which are the same whether the operation is closed or running at

100% capacity– variable costs, which may increase depending on whether more production is

done, and how it is done (producing 100 items of product might require 10 days of normal time or take 7 days if overtime is used. It may be more or less expensive to use overtime production depending on whether faster production means the product can be more profitable). [6]

Cost: Overhead Costs

• Quantitative KPI

• Frequency: Annually

• Overhead costs for a business are the cost of resources used by an organization just to maintain its existence. Overhead costs are usually measured in monetary terms, but non-monetary overhead is possible in the form of time required to accomplish tasks.

• Examples of overhead costs include:– payment of rent on the office space a business occupies– cost of electricity for the office lights– some office personnel wages [5]

Cost: Environment Utilization

• Quantitative KPI

• Frequency: Quarterly

• Measures utilization of tools and environments – a low value indicates that many tools are not used that often

• Low utilization may indicate that tools should instead be rented for the short period of time they are used, or could perhaps be used by someone else as well, since the tools are available

• Low utilization may be necessary in some organizations, but should always be considered a waste

Efficiency: Fixed Issues

• Quantitative KPI

• Frequency: Monthly

• Measures the number of issues found by the test organization that are actually fixed

• Issues that are not fixed might have informational value and will then be captured in the Stakeholder Satisfaction KPI, but otherwise hold no actual business value

• It could also be good to measure Fixed Issues / Submitted Issues, to see how many of all the issues submitted that are fixed, since there are costs associated with each submitted issue in form of reporting and analysis

• It is possible to translate the number of fixed issues to a monetary value in form of return rates and loss of sales that would have occurred because of those issues, but it requires some effort

Efficiency: Issue Leakage

• Quantitative KPI

• Frequency: Monthly

• Measures how many issues that were not found during testing and thus unknown to the product project, that was later found by customers

• If it is reported by customers it is important to the customers and should have been found during testing

• It is possible to translate the number of leaked issues to a monetary value in form of return rates and loss of sales that occurred because of those issues, but it requires some effort

Efficiency: Stakeholder Satisfaction

• Qualitative KPI

• Frequency: Quarterly

• How good is the test organization at providing the correct decision material and information to it’s stakeholders?

• This should be measured by a survey that goes out to all stakeholders

• If the stakeholders perceive that the test organization provides value to the company by the reports and decision material generated this KPI will indicate that

• This is harder to quantify in monetary terms

Organization: Leadership

• Qualitative KPI

• Frequency: Annually

• There are many ways to measure leadership qualities in an organization, more or less complex

• One way is to conduct 360 degree assessments with the employee’s subordinates, peers, and supervisor(s), as well as a self-evaluation [7]

Organization: Employee Satisfaction

• Qualitative KPI

• Frequency: Annually

• Affective job satisfaction is the extent of pleasurable emotional feelings individuals have about their jobs overall, and is different to cognitive job satisfaction which is the extent of individuals’ satisfaction with particular facets of their jobs, such as pay, pension arrangements, working hours, and numerous other aspects of their jobs. [8]

• While cognitive job satisfaction might help to bring about affective job satisfaction, the two constructs are distinct, not necessarily directly related, and have different antecedents and consequences. [8]

• One way to measure this is to let all employees answer a suitable survey

Organization: Innovation

• Quantitative KPI

• Frequency: Quarterly

• This measurement should cover both intellectual property and innovation in general

• You could focus on patent applications and invention disclosures …

• But 90 to 95 percent of all patents lack any market relevance and 99 percent fail to bring any profit to the firm (Stevens & Burley, 1997). Still, patenting is often necessary for many firms to keep their competitive advantage so clearly patents indicate something about the innovativeness of a firm. [9]

• Innovation in general is more difficult, but there are a number of measurement methods with different advantages and drawbacks [9]

Cost vs. Value

Operational Costs

Cost of Issue LeakageValue of Stakeholder

Satisfaction

Value of Fixed Issues

Analysis Cost of Submitted Issues

Target Levels

• Target levels for all these KPI will of course vary greatly between organizations because of different pre-requisites

• It is important the set reasonable target levels to motivate and drive the right behaviour

• Example: In some test organizations it may be ok to have a low number of fixed issues, as long as the stakeholder satisfaction is high, because the objective of that test organization is not to find bugs, but to validate and provide decision material

KPI Analysis

• When KPI deviate from target levels an analysis of the situation is critical

• Sometimes it may be warranted to deviate from previously set target level

• It is critical to NOT start corrective actions until the situation has been analyzed properly

• The KPI value alone does not give a complete picture – it gives indications if things are according to plan or not

Summary

• Keep it simple!

• Measure added value to the company, not internal optimization and company politics!

• Measure the right thing for the right reasons!

• Know when to use qualitative and quantitative measurements!

Reference

[1] Qualitative Propertyhttp://en.wikipedia.org/wiki/Qualitative_property[2] Defining KPIhttp://www.kaizen-analytics.com/2008/11/defining-actionable-business-driven.html[3] Measuring the Right Thinghttp://www.qualitydigest.com/inside/quality-insider-article/are-your-business-metrics-measuring-right-thing.html#[4] Occam’s Razorhttp://en.wikipedia.org/wiki/Occam%27s_razor#Probability_theory_and_statistics[5] KISShttp://en.wikipedia.org/wiki/KISS_principle[6] Operating Costshttp://en.wikipedia.org/wiki/Operating_cost[7] 360 degree feedbackhttp://en.wikipedia.org/wiki/360-degree_feedback[8] Job Satisfactionhttp://en.wikipedia.org/wiki/Job_satisfaction[9] Measuring Innovationhttp://www.innovationmanagement.se/2013/02/15/measuring-innovation-part-1-frequently-used-indicators/[10] Illusion of Validityhttp://en.wikipedia.org/wiki/Illusion_of_validity