test management for busy people
Post on 16-Sep-2014
212 views
DESCRIPTION
In today's fast-paced IT world we are often told to deliver higher quality systems to our customers under challenging time schedules, with fewer resources, and reduced budgets. As test managers and team leaders, we must become more effective and efficient with the resources we are given. We should begin questioning whether all those testing processes really must be executed and whether all that documentation should be produced, or whether some, if not all, can be streamlined. Are test plans really important? Are detailed scripts really useful? How can we create highly productive teams? Based on his upcoming book, Test Management for Busy People, Lloyd reveals how the important tasks can be determined and how time can be allocated to them, while minimizing working on multiple projects simultaneously (a real time waster). Lloyd shares a flexible framework showing which tasks can and should be streamlined so that valuable time can be restored to the busy test manager.TRANSCRIPT
TD Half-day Tutorials
5/6/2014 8:30:00 AM
Test Management for Busy
People
Presented by:
Lloyd Roden
Lloyd Roden Consultancy
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Lloyd Roden Lloyd Roden Consultancy
With more than twenty-eight years in the software industry, Lloyd Roden has worked as a developer, test analyst, and test manager for many different organizations. Lloyd was a consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden Consultancy, an independent UK-based training and consultancy company specializing in software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several countries. In 2004, he won the European Testing Excellence award.
Test Management for Busy People
A half-day workshop
Written by Lloyd Roden www.lloydrodenconsultancy.com Version 1_0 © Lloyd Roden
LRC110815
Copyright Notice
It is strictly prohibited to reproduce any of these materials in any form without prior written permission from Lloyd Roden Consultancy. This includes photocopying, scanning and the use of any recording devices.
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-1
Test Management for Busy People
1
Contents
Introduction
Doing more with less
Building an elite test team
Does it matter – does it matter much
2
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-2
Busyness is both good and bad
! busy good ! work is exciting, challenging and enjoyable ! the day goes quickly ! job satisfaction
! busy bad ! too much work to cope with ! finding stress levels are constantly high ! family suffer
are you currently “busy good” or “busy bad”?
3
Modern life wants more time…but we have less time
4
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-3
Contents
Introduction
Doing more with less
Building an elite test team
Does it matter – does it matter much
5
#1: Challenge redundant documentation
! are these test documents really needed? ! test plans ! written test reports ! test logs ! detailed test scripts ! test strategy ! test retrospective
! are these documents used or read? ! how do you know? ! what feedback do you receive? ! are they kept up to date?
if you produce documents that are not used or considered worthwhile then don’t produce them
you have 86,000
seconds in a day How will you spend them?
Don’t waste them?
6
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-4
#2: Eliminate waste
! concept of eliminating waste is good ! examples
! meetings ! emails ! duplication of effort ! documents not being used ! maintenance of the tests taking longer
than the test execution ! however…
! sometimes cutting back too much causes more problems
7
Wastage example 1: meetings
10 people @ 1 hour meeting = 10 hours • 1 person arrives 6 minutes late……………………………… • 4 people receive texts and respond taking 2 minutes…….. • 1 person gets a call and leaves room for 15 minutes……... • 6 people on laptops responding to emails…………………..
….1 hour ….80 minutes …. 25 minutes …. 3 hours
total: 6 hours wasted = 60% inefficiency suggestion: conduct meetings with etiquette
! no laptops ! no phones ! no interruptions ! start on time
8
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-5
Wastage example 2: bugs raised in error (or re-raised)
Percentage defects re-raised
Version A
TOTAL 31/96 = 33%
Percentage defects raised in error
Version A
TOTAL 23/96 = 24%
N.B: figures are based on 1 day to raise and investigate. And a cost to the company of 250€ per day
31 days
23 days
total wastage
13500€
54 days
(6% of the project)
9
#3: Don’t count test cases…
…without knowing the quality of them ! ! quantity without quality is meaningless
! example, in the last hour of testing: ! tester 1 has run 30 test cases ! tester 2 has run 300 test cases ! tester 3 has run 2 test cases ! tester 4 has run 0 test cases
! know the quality of the tests ! test the tests regularly ! review the tests ! check on coverage
the only number with meaning is
zero
10
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-6
#4: Provide some slack
! maximum velocity does not equal maximum productivity August 2010: productivity in the United States unexpectedly decreased in the second quarter
after employers expanded the workweek by the most in four years. Employees output
decreased by 0.9% per hour of output
7
4
5 2
6 1
8 3
9
efficiency is improved by 11.1% !
BUT we cannot play the game… productivity is reduced…to zero "
source: Tom DeMarco, “Slack”
7
4
5 2
6 1
8 3
11.1% spare capacity Car engines work at
their optimum efficiency (mpg) at 56mph
Management’s decision:
to fill spare capacity to increase productivity
Productivity declines unexpectedly as US workweek lengthens
11
People are not a fungible resource
money is fungible:
net result: still have £100
but people are not fungible:
net result: at least 30% loss of productivity “on average there will be a net loss of 15% productivity due to task switching”
bank 1
£100
bank 1
£30
bank 2
£50
bank 3
£20
100% on project A
30% on project A
50% on project B
20% on project C
therefore be careful of the saying “women can multi-task and men can’t”
decide to divide money into 3 banks
management decide to divide your time into 3
projects
12
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-7
Case study: Personal Assistants – what a waste!
! remember PAs: the manager’s ultimate resource
from this assigned to one manager available all the time able to do anything anytime cheaper resource BUT only 60% productive
net result: was not able to photocopy document for manager so they had to
photocopy 30 copies…and this is efficient and cost effective?
to this reduce PAs by 40% shared resource 100% productive
13
Companies who have introduced “slack”
Gore 10% dabble time 3M 15% think time Google 20% anytime Atlessian 20% Fedex days
14
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-8
when would an electric paintbrush be beneficial? when would a manual paintbrush be beneficial?
manual versus automated
but using both
requires skill and manual
intervention
#5: Tools are to “support” testing
15
Use tools wisely
! tools can help increase productivity and staff morale ! using tools requires new skills ! tools can automate tedious/boring tasks ! tools are less likely to make mistakes ! some tools can find bugs quicker ! tools can help with testing that cannot be achieved
manually ! tools can decrease productivity and staff morale
! tools do not make good testers ! too much automation can lead to poor testers ! a fool with a tool is still a fool ! automating rubbish produces faster rubbish ! tools are software and have bugs in them
16
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-9
#6: Take control of your day
! learn to say “yes” to yourself occasionally ! saying “yes” to others ultimately means saying “no” to you
! don’t allow others people to control your day all the time ! this leads to frustration
! learn to control unplanned activities ! think before reacting
! set yourself daily goals and achieve them ! John, Susan, Nora and George
17
#7: Less scripted more exploratory testing
a problem with scripted testing A
B 18
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-10
Dangers of detailed scripts
! reasons to write detailed scripts ! describes detailed tests for inexperienced testers ! automation ! provide an audit trail
! risks ! boring to write…boring to run ! more time spent writing than running tests ! maintenance nightmare ! false confidence in the quality of the
software ! your best testers will probably leave
suggestion: introduce exploratory testing sessions and limit the number of detailed scripts produced
19
! tests are prioritised against the product risks
! highest priority risks are mitigated first ! testers will use appropriate tools and
techniques to mitigate the risks efficiently
! re-prioritisation of the tests will be initiated by the re-prioritisation of the risks
! product risks will be monitored and reported to stakeholders on a regular basis
#8: Risk based testing is a must
20
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-11
Determining the “cut off”
Product Risk Probability Impact Exposure Risk Value Poor response times High High 9 90 Security violation High High 9 90 Tax calculation incorrect Medium High 8 80 Holiday pay not included High Medium 7 70 Bank Transfer interface problems High Medium 7 70 User interface may not be liked Low High 6 30 New employees not paid Medium Medium 5 25 Unable to delete accounts Low Medium 3 3 Reports are unaligned Medium Low 2 2
provides a priority list and a focus for testing
21
#9: Personal daily retrospectives
! take time to reflect ! 15 minutes at the end of the day
! what did you do well ! what could you improve on ! what did people appreciate
! take time to change ! to be the best you can be ! improvement starts with “I”
! take time to learn ! you are never to old !
true nobility is not being better than anyone else but being better than you used to be
22
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-12
#10: When “speed up” really means “slow down”
! this is only efficient if the workload remains constant
! if priorities change or workloads increase this might cause unpredicted consequences:
me
John
Sue
Amy
flow of work
e.g. John needs information from me urgently, therefore my workload/priorities change to fulfill the expectations. The net effect is that Sue becomes less busy, but she knows the “hurry up” mantra which to her means “stay busy”, so is Sue is becoming less busy, how does she show others that she is busy?
Sue slows down!! 23
Contents
Introduction
Doing more with less
Building an elite test team
Does it matter – does it matter much
24
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-13
#11: Create an “elite” team of testers
! a team that knows, understands and can apply… ! systematic test design techniques ! exploratory testing ! bug hunting ! flexibility ! creativity
! a team that will want to ! collaborate ! increase their knowledge and expertise ! adapt to the circumstance
lead by example, coach, mentor and work with your testers
a team whose “whole” is greater than the sum of
its parts
25
#12: Use the Tester’s style analysis questionnaire
things to note ! there is no right or wrong answer ! try not to think too much ! it helps us to assess our strengths
! can be used with other psychometric questionnaires (e.g. Belbin, Myers/Briggs, 16PF)
! two parts ! answer questionnaire and ! plot the values on the graph
26
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-14
The questionnaire - how to complete (x axis)
Friendly Approachable Casual Open Unstructured Social Intuitive Random Warm Perceptive
Formal Retiring Business Like Guarded Organised Introvert Logical Focused Cool Insensitive
X - Axis
x
x x x
x x
x x
x x
627
The questionnaire - how to complete (y axis)
To the point Challenging Quick Insistent Lively Impatient Adventurous Confronting Competitive Strong Minded
Indirect Accepting Leisurely Thoughtful Relaxed Patient Cautious Receptive Co-operative Analytical
Y - Axis
x
x x
x x
x x x
x x
228
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-15
Plot your score on the grid
X (6,2)
29
The Model
The Pioneer
The Facilitator
The Pragmatist
The Analyst
30
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-16
#13: Test the testers/tests
testing is quality control on development
BUT…how good are your tester/tests? Who is performing quality control on the testers?
developers bug seed, plant 100 bugs and see how many are found
provide an application for new testers with known bugs and see how good they are at finding them
we deserve the best…mediocracy is not an option 31
Areas where this idea can be used
! testing your regression tests ! manual or automated ! discover how effective they are
! testing the outsourced teams ! let’s wake up! ! discover whether they are cost effective
! testing new testers wanting to join the team ! aim to recruit the best ! quality gate into testing
32
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-17
#14: learn to be a weather person
weather people… ! predict what is likely to happen using
trends and forecasting ! understand statistics and probability ! use historical data to substantiate
their synopsis ! understand correlation between
systems ! are becoming more accurate ! don’t always get it right
! not always liked…but it is not their fault!
! but we still use them to plan and implement our activities
however…
33
Learn to predict the “projects” weather
! to predict what is likely to happen at an early stage
! to use basic statistics and probability
! to gather and analyse historical data
! understand the correlation between graphs
! to become more accurate
BUT: you wont always get it right, you wont always be liked, but it’s not your fault
0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10
0
10
20
30
40
50
1 2 3 4 5 6 7 8 9 10
34
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-18
#15: Tailor your test documentation…. and use modern technology
don’t be boring make it exclusive
challenge quality and quantity of what is produced exploit modern
technology
35
Make it relevant and make it personal
! providing information is a major aspect of testing and must be… ! accurate ! timely ! predictive ! factual ! objective ! personal ! relevant
CEO’s Dashboard
Project Manager’s Dashboard
One size doesn’t fit all 36
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-19
#16: Understand data correlation before reporting
…plus basic data significance example 1: defects found
example 2: estimated versus example 3 effectiveness of testing actual quality
good or bad?
what about test effectiveness after 1 month of live Project 1: DDP = 85% Project 2 : DDP = 60% Project 3 : DDP = 90% Project 4 : DDP = 10% conclusion: reward team 3, punish team 4? why don’t we have a look at the numbers
good or bad?
37
#17: Learn the tools of the trade
…and become skilled at what you do ! test techniques
! state transition, pairwise, domain analysis etc. ! the system
! application, architecture etc. ! test tools
! know what is available ! practice makes perfect
! test on a regular basis 38
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-20
#18: Stand out and be different
what have these stores got in common?
what have these stores got in common?
39
Create a unique selling point…
! what makes you/your team different from others… ! that are cheaper? ! that are faster?
! your USP is so that you thrive…not just survive ! example USPs
! recognising and eliminating waste ! being able to find those “nasty” bugs quickly ! measuring and reporting the value of testing ! having a positive attitude at work ! collaborating always ! constantly improving efficiency ! estimating quality… 40
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-21
#19: Apprentice train
! benefits for the apprentice ! learn from others
! hear (forget) ! see (remember) ! do (understand)
! fast track ! learning curve is shorter
! benefits for the mentor ! rewarding to see progress ! improve ourselves
! learn from the apprentice ! invest time in people
! greatest contribution we can make
41
Let’s reminisce…
! think back to the first time you were apprentice trained ! cooking with your mum? ! home improvement with your dad? ! washing up (or nowadays - loading
the dishwasher!) ! what type of experience was it?
! good/bad/exciting/non-event
42
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-22
what they are ! opportunity to learn and
increase skills ! a safe environment to
make mistakes ! a framework to prove
yourself worthy of the current/next job role
! an environment to provide constructive feedback
! to be taken seriously
what they are not ! errand running ! a career trap " ! a guarantee ! an excuse to make lots
of mistakes ! a training course/
certification scheme, all theory and no practice
Apprenticeships: what they are/are not
43
Chief Tester one person is responsible for mentoring all new staff
Hierarchical
you mentor subordinates and you are mentored by your manager
Buddy pairing testers to mentor each other, accountability is encouraged
Coterie
everyone mentors everyone!
Various models to consider
44
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-23
“An excellent tester this one is”
! a “jedi” tester will… ! let themselves be distracted by anomalies
and new ideas ! encourage variability, avoid repeating the
same test twice ! use their confusion as a resource ! know their limits and get to know/use
subject matter experts ! adapt testing to fit the context ! develop resources and tools ! train their minds to be curious, cautious
and critical ! take notes and report results in a useful
and concise way 45
#20: Take time to sharpen your axe
! don’t procrastinate ! create tools and utilities to make testing more efficient ! share your ideas with others ! understand new methodologies and technologies and how
they will affect you (keeping one step ahead) ! measure your own effectiveness and improve ! create ‘me’ time (kick the leaves) ! try something new ! become a champion…
complacency is only one step ahead of
apathy make it your mission to become the
best at what you do
46
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-24
Contents
Introduction
Doing more with less
Building an elite test team
Does it matter – does it matter much
47
#21: Choose your battles
! you can’t fight every battle… so what is important? ! career progression for testers ! remuneration ! providing slack ! a good working environment ! fairness ! more resources ! advice given is used ! purchasing test tools ! efficiency ! a good test environment ! test estimates being agreed
48
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-25
#22: Don’t worry about test estimation…too much
! understanding the problem with test estimation ! recognising that we can test in less ! provide options using a variety of techniques
Testing 4 weeks
Testing Testing Testing 2 weeks 1 week
49
#23: Estimating quality is a must
fact: estimation is based on 2 aspects
effort + people = schedule half truth for testing:
effort + people + quality = schedule we must also estimate the number and severity of the bugs we are going to find to make our estimates meaningful (a direct correlation)
WARNING: be careful of Brook’s law – 9 women cannot have a baby in 1 month! Some activities cannot be accelerated by adding more resource
50
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-26
Estimate, monitor and report quality
how to estimate quality ! guess ! use historical data (previous projects, defect density) ! use models
what to monitor and report ! actual versus expected ! trend analysis and forecasting ! correlation between graphs
0
10
20
30
40
50
1 2 3 4 5 6 7 8 9 10 0
50
100
150
200
250
1 2 3 4 5 6 7 8 9 10
Cycle #
Actual bugs
Masked rate
Observed bugs
Fix Rate
Fixed bugs
Insert rate
Inserted bugs
Remaining bugs
1 10 30% 7 80% 6 20% 1 5
2 5 30% 4 80% 3 20% 1 3
3 3 0% 3 100% 3 0% 0 0
4 0 full regression test
51
#24: Remuneration and career path
! remuneration must represent their skill ! recognising that testing is a skill and profession ! fairness throughout the organisation
! define a career structure for testers ! where management is not the only option ! encourages testers to aspire in their career ! essential to recruit and retain good quality testers
functional career model
management technical application
functional non-functional test consultant
test manager
team leaders
senior technician
DBA/env technician
test automator
designer designer
executor executor 52
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-27
#25: Demonstrate the ROI for testing
requirements team… has an idea and forms requirements design team… takes (vague) requirements and designs the product development team… uses the design to develop the product test team…? if testing doesn’t provide a good ROI… then it is not a good investment
YES NO
ROI
! all teams must provide the company with a good ROI, the problem is perception…
ROI must be measured in quantifiable
terms (CSPB,
DDP, DFP, PRDFP etc.)
53
#26: Provide management with feedback on decisions that they make
! example: 4 weeks of testing:
! new data obtained:
0
20
40
60
80
100
120
0 1 2 3 4
Number of bugs
Actual
Management want to implement the software, but you suggest a
further week of exploratory testing and a final week of regression
testing due to the high number of bugs that have been found
0
20
40
60
80
100
120
140
0 1 2 3 4 5 6
Number of bugs
Actual
25 new bugs found and fixed…full regression pack run
54
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-28
Using data to report back
# cost of bugs to find and fix in test = 1000€ # cost of bugs to find and fix in live = 3000€ # 1st month of live operation customers find 10 bugs ! if the old date (4 weeks) was used
! assume 35 bugs found by customer ! DDP = 100/135 = 74% ! cost of fixing = 35 * 3000€ = 105,000€
! if the new date (6 weeks) was used ! DDP = 125/135 = 93% ! cost of fixing = 10 * 3000€ = 30,000€
+ 25 * 1000€ = 25,000€ = 55,000€
cost saving 50,000€
DDP increase 19% Therefore a good
decision by management !
55
#27: Make your test environment the strongest link ! often the test environment becomes the
weakest link ! concentrate too much on test cases and
test scripts ! reliant on other people ! often difficult to set up well ! we believe the environment will “just” happen
! make it the strongest link ! constantly think about your environment ! make it a priority prior to test execution ! learn to recognise environmental issues
and raise these with your managers
56
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-29
Bad environments
if your environment is wrong, out-of-date, volatile, not representative, or messy then testing cannot succeed
ignore your test environment and the bugs will breed
57
#28: Is there any value in raising bug reports
! fact: there is very little value in raising bug reports: ! during development (component/integration tests) ! if you are only concerned in fixing bugs
in an automated environment ! if you are sitting with the developer and they
can fix the bug immediately ! if you are not concerned with learning lessons
or improving the process ! if there is a negative ROI for the time it
takes to raise a bug log ! BUT…it depends on what we mean by “value”
…relative worth, merit or importance
58
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-30
your company
The value of our bug logs
negative ROI • information not used • no improvements • takes more time to raise than fix
positive ROI • help with progress • study bugs for defect prevention • testers improve testing skills • reduce wastage • data is used wisely 59
#29: Can we “go-live”? Answer is always “yes”
…however
! Test Manager/Tester’s responsibility ! to provide information to stakeholders to
make informed decisions ! to offer opinions about the quality of the
software that may/may not be agreed with ! to be the quality conscience of the
project ! to highlight warning signs clearly
yes no
60
© Lloyd Roden TMBP140308
Test Management for Busy People Workshop-31
#30: Believe in yourself…and your abilities
! low self-esteem is one of the main
contributors to lacking drive and enthusiasm: ! reignite your passion
! fan the flame before it goes out ! see change as an opportunity rather than a threat ! create good habits
! 66 days to create a good habit ! but only 25 days to create a bad habit
practice looking in the mirror each morning and saying…
“I am great!”
no one on their deathbed said “I wish I had spent more hours at the
office 61
Summary
! busyness can be both good and bad
! being busy bad requires action ! introduce various ways of
doing more with less ! build an elite test team that
you can trust and rely on ! there is a need to decide
whether certain aspects are worth being concerned about
! learn to say “no” to others more so that you can say “yes” to yourself
62
© Lloyd Roden Testers Style Analysis Page 1 of 1
X-Axis
Y-Axis
Friendly
__ Formal __ To the Point __ Indirect __
Approachable
__ Retiring __ Challenging __ Accepting __
Casual
__ Business Like __ Quick __ Leisurely __
Open
__ Guarded __ Insistent __ Thoughtful __
Unstructured
__ Organised __ Lively __ Relaxed __
Sociable
__ Introvert __ Impatient __ Patient __
Intuitive
__ Logical __ Adventurous __ Cautious __
Random
__ Focused __ Confronting __ Receptive __
Warm
__ Cool __ Competitive __ Co-operative __
Perceptive
__ Insensitive __ Strong Minded __ Analytical __
__
__
Grid 10 9 8 7 6 0 1 2 3 4 5 6 7 8 9 10 4 3 2 1 0
Based upon “The Communication Styles Analysis”
Tester’s Style Analysis Questionnaire
Name