test managers: how you can really make a difference

TP PM Tutorial

10/1/2013 1:00:00 PM

"Test Managers: How You Can

Really Make a Difference"

Presented by:

Julie Gardiner

The Test People

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Julie Gardiner

The Test People

In the IT industry for more than twenty years, Julie Gardiner has held positions as an analyst

programmer, Oracle DBA, and project manager. She has first-hand experience as a test

analyst, test team leader, test consultant, and test manager. Previously with UK-based Grove

Consultants for seven years, Julie now is head of agile, talent and transition management with

The Test People. A certified ScrumMaster and agile coach, Julie also specializes in risk-based

testing, agile testing, test management, and people issues.

1

Passionate about Testing, Excited about Technology Passionate about Testing, Excited by Technology

Test Managers: How you can really make a difference 13th August 2013 Julie Gardiner Email: [email protected] Twitter: @cheekytester

© Julie Gardiner

Passionate about Testing, Excited about Technology

What makes us valuable on project

How to keep testing valuable on projects

Showing the value of testing

Contents

2


What does a Trusted Advisor look like?


What senior managers are looking for

§  commitment not just involvement –  knowledgeable enthusiasm –  high quality teams

§  respect for their decisions –  an understanding of office politics –  an appreciation of other pressures and issues

§  honesty and courage –  eyes on the deadline but hands firmly on quality –  a realist with a hint of optimism

plus… a sense of humour

J

3


What if senior managers seem uninterested? §  lack of understanding

–  what do we do with our time? –  misconceptions regarding testing/development

§  boredom –  information overload – too much detail –  the wrong information presented (does not understand)

solu%on: educa%on

solu%on: enquire

“The bitterness of poor quality is remembered long after the sweetness of meeting the schedule is

forgotten”


Being a “Trusted Advisor” requires three core skills

building relationships Source:

The Trusted Advisor David Maister, Charles Green Robert Galford

4


Trust must be earned & deserved

§  trust grows over time rather than just appears §  beware that trust can be lost in an instant

§  trust is personal §  it involves both rational and emotional factors

§  trust is a two-way relationship §  trust is different for the test manager & senior

manager §  trust will entail risk

§  if risks are not created, trust will flounder

earning trust

to be trusted is never a right… it is always a privilege and honour © Julie Gardiner


Building relationships

§  deal with senior management as “human beings” §  formula to building a strong relationship

§  BSR = AQ + LTA §  be open and honest – not a sycophant §  when you need help – ask

–  this can be a sign of strength rather than weakness

§  show appreciation

make it your goal to be someone managers turn to first when issues arise and advice is required.

building relationships

5


Giving advice effectively §  be prepared

–  look beyond the “now” & understand the whole situation

§  advice is almost never a purely logical process –  don’t exaggerate or lie – ever! (this destroys trust) –  give good news before bad news (don’t hide bad

news) –  be fair with information

§  know your audience –  the advice we give must be helpful –  different people need different information

giving advice


Developing the right mindset

six key attitudes to consider §  focus on the other person §  being sincere §  having confidence (in oneself) §  adopting a blameless culture §  taking responsibility §  recognise strengths and weaknesses

–  in you and your team

having a right attitude and mindset is a massive step

closer to becoming a

“Trusted Advisor” to Senior

Management

6





Contents


Company Objectives?

QUALITY TIME COST

Quality

Time Cost

Priority

© Julie Gardiner

7


Managing testing

Plan Measure

Review Goal

Policy overall approach to testing

Strategy


Test policies •  represents testing philosophy for organisation

–  outlines the corporate goals for testing

•  guides to stated goals and objectives •  communicates good practice for the organisation

–  approach, culture and standards

•  developed by the IT, R&D or Product Development •  apply to both new development and maintenance •  short, static, high-level document •  some areas covered include

–  test process, levels of testing, success factors, measurements, test process improvement

8


Test policy examples

•  testing is to be risk-driven

•  reviews will include testers

•  auditable test documentation records will be kept

•  entry and exit criteria will be specified and enforced

•  the value of testing will be monitored and reported

What policy statements do you have in your organisation?


Policy, Strategy and Plans Company Policy for Testing

Company/Project Strategy for Testing

Project (High Level) Test Plan

9


Monitoring

•  regular and timely supply of information •  what is actually happening •  to compare with the plan •  right use of words, graphs and tables •  interpretation & explanation is often required •  enables effective and timely control •  used to predict what is likely to happen

now to introduce some powerful monitoring techniques to display on your

dashboard…


S-‐curves for: tests run/a@empted

tests passed/complete bugs found

bugs fixed

One: S-curves

Plot the points you know, and planned end point e.g. no. tests run so far & planned final no. at end date Plot trend line: 3rd order polynomial

Useful approximaPon: straight line

Pme

Source: Marnie Hutcheson

How to use S-‐curves

10


Example using S-Curves

Source: Tim Trew, Philips


Two: Defect Measure (DM) Rate

DM = 10*H+5*M+L DM Rate = DM/(hours of test effort)

11


Three: System Availability

0 2 4 6 8 10

Hours available in a day

Mon

Tue

Wed

Thu

Fri

Sat

Sun

Mon

Tue

Wed

Days

Host System Availability

AvailableDown time

Based on 10 hour day we lose 2 days of down time!


Four: Regression Test Coverage Release 1.0 Release 1.1 Release 1.2

Key: not run defects found no defects

Solo Personnel Solo Payroll Personnel Breadth Payroll Breadth Personnel Scenario 1 Payroll Scenario 1

Test Procedure Release 1.0 Release 1.1 Release 1.2

12


Five: Radar Diagram Rating Ranges

Code Turmoil(loc) 6 1 = 0-50, 2 = 51-100, 3 = 101-150, 4 = 151-200, 5 = 201-250, 6 > 250Defects found this week 6 1 = 0-2, 2 = 3-4, 3 = 5-6, 4 = 7-8, 5 = 9-10, 6 > 10Total Open Defects 2 1 = 0-10, 2 = 11-20, 3 = 21-30, 4 = 31-40, 5 = 41-50, 6 > 50Test Success Rate 1 1 = 100%-98%, 2 = 97%-95%, 3 = 94%-92%, 4 = 91%-89%, 5 = 88%-86%, 6 < 86%Test Completion Rate 5 1 = 100%-95%, 2 = 94%-90%, 3 = 89%-85%, 4 = 84%-80%, 5 = 79%-75%, 6 < 75%TOTAL RATING 20 5 to 10 11 to 15 >15

Raw Data 10/23/2013 17-Feb-13 24-Feb-13 03-Mar-13Code Turmoil 1127 1409 1020 799Defects found this week 23 16 6 13Total Open Defects 24 32 7 12Test Success Rate 95.50% 98.70% 99.50% 99.80%Test Completion Rate 44.80% 70.00% 71.80% 78.90%

0 1 2 3 4 5 6

Code Turmoil(loc)

Defects found this week

Total Open Defects

Test Success Rate

Test Completion Rate

Source: Mike Ennis, Managing the End Game of a Software Project, STAREast


Six: Outstanding Risks

Progress through the test plan

today end date

residual risks of releasing

TODAY Res

idua

l Ris

ks

start

Source: Risk Based E-Business Testing – Paul Gerrard & Neil Thompson

all risks ‘open’ at the start

13


Benefit (or objective) based test reporting

Open

Closed

Ris

ks

Open

Open

Closed

Closed

Open O

bjec

tive

Obj

ectiv

e

Obj

ectiv

e

Obj

ectiv

e

Bene

fit

Bene

fit

Bene

fit

Bene

fit

Bene

fit

Benefits available for release Source: Paul Gerrard

Obj

ectiv

e

Bene

fit

Closed


Seven: Reliability/Confidence Curve

Reliability/Confidence Curve

0

20

40

60

80

100

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1

Release

Tim

e (

min

ute

s)

Time to failure

alternative 1 alternative 2

R 1.0 R 1.1 R 1.2 R 1.3

14





Contents


Three key measures to consider

Defect Detection Percentage (DDP)

Pre-release Defect Fixed Percentage (DFP)

cost saving per bug

15


“Testing is expensive”

•  compared to what? •  what is the cost of NOT testing, or of bugs missed

that should have been found in test? –  cost to fix bugs escalates the later the bug is found –  poor quality software costs more to use

o  users take more time to understand what to do o  users make more mistakes in using it o morale suffers o  => lower productivity

•  what does a bug cost in your organisation?

Red Use Des Test 1

10

1000

100


How much does it cost (based on Boehm)

How much do defects cost?

02000400060008000

1000012000140001600018000

requ

irem

ents

spec

ificati

on

initia

l test

syste

m/acce

ptanc

e

live

runnin

g

FolksamBoehm

Source: Kristina Bihlar, Folksam BR

16


How much do our defects cost Defect in: Manhour to fix Costrequirements 1 665specification 2.5 1663initial test 4 2660system/acceptance test 8 5320live running 15 9975

Cost per hour in SEK: 665

What if we found 100 defects during testing:During system/acceptance test 532,000what could have been found in initial test We saved: 266,000what could have been found in analysis We saved: 465,500

But it could have cost: 997,500 …Source: Kristina Bihlar, Folksam BR


How expensive for you? •  do your own calculation

– calculate cost to fix bugs missed by testing

– calculate cost to fix bugs found in testing

– calculate cost of testing o people’s time, machines, tools

•  estimate if no data available – your figures will be the best your

company has!

17


Questions you may be asked How good is the testing anyway? Can you prove you are doing a good job?

Your testing can still be just as good in less time, can’t it? (That deadline pressure really didn’t matter, did it?)

Is the testing any better for this release? (Have we learned anything?) (Have we really improved our testing?)

How many bugs have we missed?


Some questions for you •  do you keep track of defects?

–  defects found in testing o  different test stages (e.g. system test, user acceptance

test), o  different releases (e.g. testing for an incremental

release) –  defects found in live running

o  reported by users / customers •  can you find these numbers from a previous project and your current

project? •  do you have a reasonable number of defects found?

if so, you can use DDP to measure your test effectiveness

18


Defect Detection Percentage (DDP)

•  "this" testing could be

–  a test stage, e.g. component, integration, acceptance, regression, etc.

–  all testing for a function or subsystem –  all testing for a system

Defects found by this tesPng Total defects including those found aYerwards


50 100 Defects found

after testing: Total defects found:

Release

50 0

50 45 40 35 30 25 20 15 10 5 0

Defects Found

Time

62 12 62 81

69 19 69 72

74 24 74 68

77 27 77 65

85 35 85 59

87 37 87 57

88 88 38 57

10 Defects found in testing: 42 50

Effectiveness at finding defects 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

DDP

50 DDP = % =

19


Options for measuring DDP

•  what defects? – simplest: all test defects / all defects so far – by severity level

•  how "deep" to go? (how much analysis) – deeper analysis gives more detailed information o but is it really worth the extra effort that would

be needed? •  start simple

– simple information is much better than none –  learn from what information you have


DDP versus DFP

Defects found

and fixed

Defects found aYer release

Defects found

and not fixed

Testing

Pre-‐release Defect Fix Percentage =

defects fixed before release

all defects found before release

Defects found by tesPng all defects found

Defect DetecPon Percentage =

20


Top 10 common responses from the survey

“what do senior managers actually look for in a test manager?”


Top 10 surprise responses from the survey “what do senior managers actually

look for in a test manager?”

21


Ways forward for you

•  set yourself an agenda – choose most important improvement areas – be practical – plan the assessment of the changes

•  plan actions and dates – assign responsibility – how to support the initiative –  timetable for improvement milestones

•  the value of today is what you do next

Passionate about Testing, Excited about Technology Passionate about Testing, Excited by Technology

Questions

?

Phone: +44 (0)113 8151779 Email: [email protected] Web: http://www.thetestpeople.com © Julie Gardiner

test managers: how you can really make a difference

Technology

testing valuable

agile testing

testing philosophy

levels of testing

testing guides

riskbased testing

test management

test analyst