amibug.com, inc. © robert sabourin, 2008september 15, 2015slide 1 toward an exploratory testing...
TRANSCRIPT
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 1
Toward an Exploratory
Testing Culture
Robert Sabourin
President & Principal consultant
AmiBug.Com, Inc.
Montreal, Canada
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 2
Toward an Exploratory Testing Culture
Traditional testing teams often agonize over exploratory testing. How can they plan and design tests without detailed up-front documentation? Stubborn testers may want to quit because they are being asked to move out of their comfort zone. Can a team’s testing culture be changed? Rob Sabourin describes how several teams have undergone dramatic shifts to embrace exploratory testing. Learn how to blend cognitive thinking skills, subject matter expertise, and “hard earned” experience to help refocus your team and improve your outcomes. Learn to separate bureaucracy from thinking and paperwork from value. Explore motivations for change and resistance to it in different project contexts. Leverage Parkinson’s Law—work expands to fill the time available—and Dijkstra’s Principle—testing can show the presence of bugs, but not their absence—to inspire and motivate you and your team to get comfortable in the world of exploratory testing.
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 3
– Defined– Philosophy
• Fundamental Question• Dijkstra’s Principle• Parkinson’s Law• Covey
– Purpose– People– Learning– Context– Diverse Testing Ideas– Case Study
Toward an Exploratory Testing Culture
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 4
Exploratory Testing
Defined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 5
• Concurrent– Learning– Design– Execution
ET
Def
ined
Toward an Exploratory Testing Culture
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 6
Exploratory Testing
An Example:
Bug Confirmation Testing
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 7
Bug Confirmation
• Confirm that the “fixed” bug is really fixed
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 8
Bug Confirmation
• Was bug fixed?
• Did bug move?
• Did developers accidentally inject a new bug?
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 9
Bug Confirmation
Was bug really fixed?1. Use repro steps from bug report
2. Confirm bug was in old build
3. Show bug is not in new build
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 10
Bug Confirmation
Explore: Did the bug move?1. Exercise related capabilities
2. Vary steps to reproduce
3. Investigate features that share code
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 11
Bug Confirmation
Explore: Did developers inject a new bug?1. Exercise application capabilities
2. Vary data
3. Walk through scenarios
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 12
Bug Confirmation
Jason’s Approach?1. Speak with developer2. Confirm bug was in old build (repro)3. Confirm bug is not in new build (repro) 4. Identify a usage scenario which would have
exposed the bug5. Walk through the scenario many times with
variation of steps and user data6. Stop when perturbation is absurd
ET
Def
ined
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 13
Exploratory Testing
Philosophy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 14
Fundamental Question
• How do you know when you are finished?
Phi
loso
phy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 15
Edsger W. Dijkstra
• “Program testing can be used to show the presence of bugs, but never to show their absence”
Phi
loso
phy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 16
C. Northcote Parkinson
Parkinson’s Law:“…work expands so as to fill the
time available for its completion…”
Phi
loso
phy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 17
“…begin with the end in mind …
“…first things first …"
Steve Covey
Phi
loso
phy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 18
"No! Try not, Do. Or do not.
There is no try."
Yoda
Phi
loso
phy
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 19
Exploratory Testing
Purpose
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 20
Purpose of Testing
• Common definition:– To find bugs before our customers do!
• Broader definition:– The role of testing is to provide objective input to
facilitate business decisions!– Keeps stakeholders aware of all issues or
concerns that relate to shipping a product!
Pur
pose
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 21
Bug Defined
• To make our job more fun, whenever we have a concern with software, we call it a “bug”.
Pur
pose
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 22
Exploratory Testing
People
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 23
• It’s all about people! (and the occasional bug too)
Peo
ple
Toward an Exploratory Testing Culture
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 24
Leadership
• Different Strokes for different folks at different times!– Adapt leadership style to the situation– Choose leadership styles deliberately!
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 25
Leadership
• Basic Leadership Styles• Directing• Coaching• Supporting• Delegating
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 26
Leadership
• Directing Leadership• Tell people specifically what to do• Provide constant feedback, praising and
redirection• Used when someone is new to a task and
uncertain as to how to successfully achieve the task
• Used sometimes in an emergency situation
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 27
Leadership
• Coaching Leadership• Provide guidance and advice on how to achieve
goals • Does not need close direction but needs to learn
how to achieve success• Used when someone has a proven track record
but is new to this specific task• Team member is mature enough to ask for
assistance
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 28
Leadership
• Supporting Leadership• Staff participate in decision making with leader• Staff works with leader to establish goals and
milestones• Person can work quite autonomously but needs
leaders help• Team member is mature
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 29
Leadership
• Delegating Leadership• Staff is given broad goal and takes full
ownership of task• Constant feedback is not required• Confirm big picture business drivers have not
changed• Team member is autonomous
Peo
ple
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 30
Exploratory Testing
Learning
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 31
Unprepared
Learning
Lear
ning
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 32
Just-In-Time Testing
Sharpen Testing Skills• Thinker• Detective• Reporter• Diplomat• Negotiator• Cheer Leader• Pragmatist
Lear
ning
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 33
Exploratory Testing
Context
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 34
Context Drivers - BTO
• Business– Value– To whom?– Why?
• Technology– Solutions
• Organization– Corporate Structure– Team Structure– Roles and Responsibilities
Con
text
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 35
Con
text
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 36
Exploratory Testing
Diverse Testing Ideas
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 37
Testing Ideas
• Collect testing ideas• From testing ideas build a
series of testing objectives– Each can be assigned as
work to testers– Each can include all, part of,
or multiple testing ideas
Div
erse
tes
ting
idea
s
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 38
Testing Ideas
• Investigative approaches– We become truffle
snorting pigs and try to find useful information in all evidence we discover
– We can even get good ideas from out of date sources
Div
erse
tes
ting
idea
s
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 39
Test Idea Sources
• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies
Div
erse
tes
ting
idea
s
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 40
Test Triage
• Allocate Testing Assignments to Testers– Make sure testers know context– Best thing to test– Best person to test it– Best people to explore it– Best lead– Are subject matter experts required
Div
erse
tes
ting
idea
s
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 41
Thank You
Questions?
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 42
Out of the Frying Pan into the Fire
The case of the distracting documentation
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 43
The case of the distracting
documentation • Corporate goal
– respond to market pressure without destroying ongoing development projects
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 44
The case of the distracting documentation
• Before– In Testing
• Heavily scripted • Subject Matter Experts• Independent team of
system testers
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 45
The case of the distracting documentation
• Before– In Development
• Teams torn apart to react to emergencies
• Heroic Knights in shining armor• Development uses several levels
of documentation• Tons of "integration testing”
before release to test team• Inconsistent unit testing
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 46
The case of the distracting documentation
• Before– In Product Management
• Many different roles managing work• Many different requirement lists• Dominant consensus management from
highest levels
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 47
The case of the distracting
documentation • Setup
– Urgent project selected as pilot– Storyboards replaced
traditional requirements– Assigned team members by
surprise– No training (wing it)
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 48
The case of the distracting documentation
• Problems– Testing team very uncomfortable– Discovered exploratory testing– Tons of overtime– Product management
not active– Treated as a heroic effort
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n
AmiBug.Com, Inc.
© Robert Sabourin, 2008April 19, 2023 Slide 49
The case of the distracting
documentation • Remedy
– Active product management– Consolidate backlog– Train team in method– Involve testers in planning sessions– Coaching on exploratory and Just in time
testing– Consistently use unit test framework– Teach team how to use "stories" for
requirements and testing– Team was committed during iteration
Cas
e of
the
Dis
trac
ting
Doc
umen
tatio
n