Realistic Test Process Improvement by Professionals:

A Case Study

Brian Wells,Experimentus,

UKW14

Realistic Test Process Improvement by Professionals: a Case Study

Brian WellsExperimentus

UK

www.experimentus.com

Session W14EuroSTAR 2007 5 Dec 2007

What will we cover?

Test Process Improvement – what are the issues!

The case study organisation: today’s example!

The (recent) Past

Current Improvement activities

The main “Enablers”

Professional resources as well as a Test Profession!

Communication, culture change & skills/knowledge

acquisition

Measurement!

Test Process Improvement Programme Customer Survey

How many of you have been part of process improvement

programmes?

How many programmes have said they will do things so much

better?

And save lots of money/effort?

How many of you have seen PREDICTED, DEMONSTRABLE,

CONTINUING savings/improvements?

Why do Test Process Improvement Initiatives Fail…

Usually becomes “shelf life” after a short

period as people do not understand why and what

the benefits are

Senior Mgt fully supports change but difficult at

operational level

Lack of clear direction AND acceptance

/understanding of the changesat all levels

Who really understandstest

Management?

Inconsistent test skills within organisation

Lack of Acceptance of testing

as a profession and a formal discipline

Process Improvement often driven by local “heroes”

Test Process Improvement Initiatives Fail……

Insufficient effort to ensure the right professional resources are

placed in a structured profession

Full Communication, awareness, training, mentoring and monitoring

of implementation

Inaccurate monitoring, control and measurement of benefits

accruing to all levels

Less than effective, efficient and continuing change

Retail IT – HBOS

A case study

The Case Study Organisation

Major UK Banking & Financial institution

4 Business Divisions; Retail Banking the division under the spotlight

today with offices in Leeds, Halifax and Cardiff

Retail IT covers all technology deliverables (mix of on and off shore

resources for development and increasingly test)

Central Test Authority with ownership of test processes etc within

Business Division

Head of Central Testing plus 4 permanent resources from different

backgrounds

Owns the test methodology and standards

Improvement driven by need to be smarter, more efficient and reduce

costs while improving quality!

Retail IT, HBOS – The Starting Point

Formal TPI® assessment undertaken in 2004/2005

Results and the input from Retail IT defined objectives for 2-

year Test Improvement project

Central Test Authority managed implementation of specific

activities to:

Standardise Reporting

Implement Test Metrics Programme

Implement Career Paths & Test Organisation

Undertake Test Life Cycle Improvements

Improve Test Data Management and make visible

Consolidate Test Tools Inventory & Best Practice

Overall objective was to ensure Test is <=40% of total project

What Progress?

Commissioned a TMM Assessment in Q3-Q4, 2006

Evaluated competitive TMMi® and TPI® bids to undertake

appraisal of Test and Test-related Quality Process Maturity

TMMi appraisal to Levels 2 & 3 demonstrated best combination of:

Professional, Industry recognised, Assessment approach

Speed (of appraisal and elapsed time)

Breadth (of appraisal across organisation)

Alignment to Organisation going forward

Transparent to all

Structured delivery of Findings and recommended, prioritised

Process Improvement Plan

TMMi Assessment Objectives

Objectives of the TMMi Assessment exercise were to:

Have the 2-year project process improvements delivered as

defined?

Accurately and succinctly measure current Test Process capability

Validate priority of their existing planned improvement activities

Identify and prioritise any gaps

Identify what's been achieved, what now needed to be done, and

why

In other words…..

Doing the right things?

Doing them in the right way?

The Results – The original 2-year Project Objectives

General feeling was that they had succeeded

TMM Appraisal demonstrated that most 2-year Project

deliverables had been broadly achieved except:

Test Automation (and associated Regression Test Guidance)

Standard approach to risk-based testing

Real progress on alleviating issues around test environments

& data

Estimate 75% of Project deliverables implemented

BUT BASE ACCURATE MEASUREMENT OF ACHIEVEMENT OF

BENEFITS IN INFANCY!

The Results – TMMi Assessment

Appraisal findings indicate achievement of approximately 70%

for level 2 and 75% for level 3

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2.1-T

DGP

2.2-T

PE

2.3-T

TM

3.1-T

O

3.2-T

TP

3.3_TIS

DLC

3.4-C

MTP

KPA Maturity Achievement

Partially/Not Deployed

Mostly Deployed

Actively Performed

What was changed

Approach to Implementation

Change of emphasis - Central Test Authority manage and

control all change - Business as Usual now

Identified 68 specific, detailed improvement opportunities

grouped into strategic Improvement Work Packets

Improvement Work Packet delivery process defined as:

Definition of Process, Procedures, Templates

Structured Reviews andWalkthroughs

Defined specific training modules

Run pilots and evaluate Phased implementation

Strategic Work Packet 1 - Stronger Policies & Framework

Aligned with TMM

Clear mandatory direction from top

Clear Exemption procedures

Consolidated test framework elements by:

Restructuring Intranet Portal

Defining clear integration into Test and

delivery Life Cycles

Ensuring it is easy for all to understand ,

access and use!

E2E Test Life Cycle with clear Test Quality

Gates

Strategic Work Packet 2 - Test Deliverable Matrix

Test Deliverables Matrix

List of Testware with Owners, Reviewers and Informed roles

Covers entire Test Life Cycle activities

Mandatory and Optional deliverables linked to Test Life Cycle, project

size etc with integrated project controls

Linked to Test Quality Gates, Structured Reviews

Benefits accruing from these three Work Packets are:

Clarity on mandatory, standard, integrated framework for all Business

Areas – removed confusion

Easy to understand, follow and use – streamlined effort and removed

duplication

Critical enablers to realise tangible benefits

Strategic Work Packet 3 - Structured Reviews Procedures

Fully defined Reviews Procedures and Templates which:

Covers informal and structured reviews

Gathers critical metrics data

Includes full training programme

Fault Distribution Curve

0

50

100

150

Test Stages

Faults

Current

Improving

Target

Benefits are:

Improving early fault

detection and removal!

Reduction in the cost of

fixing faults in execution

Strategic Work Packet 4 - Risk Based Testing

Risk Analysis

0

39

78

0 39 78

Liklihood

Imp

act

SMS Server

Interface 1

Database1

Web Server 1

Data Archive 1

Telephony S1

Broadcast 1

Device 1

Implemented robust approach to Product Risk

using a Boston Square approach

Benefits are:

Identification of areas of high risk

early and first

Critical fault detection and resolution

earlier

In parallel, reducing test effort overall

“down stream”

Greatly improved communication and

understanding between all roles on

project

Strategic Work Packet 5 - Measurement Programme

Defined and published detailed Test & Debugging goals

Standardised test management and fault tracking structure, work flow

Ensured common reporting and analysis at test activity/project level

Implemented formal Test Quality Gates and enhanced, minimum

mandatory Entry/Exit criteria

Formalised Central Test Authority Process Audit/Quality Assurance

responsibilities within programme

Improvement activities capable of being measured!

Measurement Objectives & Benefits

Approved Test & Debugging Goals provide specific objectives/targets to:

To improve Test Efficiency & Effectiveness

Measure accruing, anticipated benefits of process improvements

Accruing benefits are:

Developing accurate picture of where they are today

Various elements already leading to more efficient and effective planning

through to implementation

Identifying additional opportunities for improvement already

High % of faults Rejected as “Not a Fault”

High % of faults Re-Opened (i.e. not fixed 1st, 2nd and more times)

Large test effort finding few faults

Test Process Improvement by Professional Testers!

Profession versus Professional?

“Profession” defined as “a paid occupation, especially one

involving training and a formal qualification”

“Professional” defined as “worthy of or appropriate to a

professional person; competent” – “a person having impressive

competence in a particular activity”

The main enabler identified was (radically) to:

Create and embed a Test Profession career structure

AND

Use these dedicated, increasingly skilled and professional test

resources to be instrumental in ensuring robust, long term

implementation of test process initiatives by Professionals

The Rationale

A Profession can be created; becoming a Professional takes time!

Organisational culture change is easy to impose from above but hard to

get working operationally

To address these, the key enabler identified to “plant” key test

professional in every business area:

Single, qualified point of contact: easy communication and manage

culture change effectively

Effective link between Method and practise

Ensure test integration with all other elements of SDLC

Streamline implementation and early, continuous benefits process

improvement initiatives

The Progressive Approach

Test career structure institutionalised within HR

ISTQB Foundation Qualifications as part of minimum Test skill set

Area Test Champions appointed and Testers Forum active

Pressure for Area Test Champions to be Central Test Authority

“lieutenants”

Responsibility for operational implementation of all aspects of

testing – including change

Test resources maturing skill set and reputation/respect

Enhanced training capability linked to Test Process Improvement Work

Packets

Organisation actively accept and embrace dedicated test professionals

AND CHANGE

The Delivery Stream Test Managers

Intended to enhance Test Organisation capability

Highly skilled: both in Test and “soft”

Primary responsibilities are to:

“Own” Testing within Delivery Stream

Provide Delivery Stream Test Management

and communication

Ensures early and continuous test

engagement

Other benefits being realised

Greatly improving communication, awareness and acceptance

of all things test!

Monitoring and enhancing maturing test skills and capabilities

Represents and promotes Professional testers and testing

Facilitates internal, standard processes implemented in

controlled and robust way

Culture change much reduced “blocker”

Allowing other enabler process changes to be implemented

quicker, more effectively and robustly

Ensuring changes, and the demonstrable measurement of

change, happens at all levels

The Key Message!

Evaluated “traditional” approach to implementing change and common

problems

Central Test Authority identified key enabler of effective, efficient, robust

change: their Test Professionals!

Identified, trained Delivery Stream Test Managers highly experienced in

Test and soft skills

Change is removing common problems encountered with change!

It is Working!

Any Questions?

Thank You!

Brian Wells:

www.experimentus.com

Brian.wells@experimentus.com

+44 (0)7725 709262