rapid testing presentation
TRANSCRIPT
-
8/11/2019 Rapid Testing Presentation
1/46
MA
// 0
A Rapid Introduction to Rapid
Software Testing
Presented by:
Michael BoltonDevelopSense
0 , 00, , 0
0 00 . ..
-
8/11/2019 Rapid Testing Presentation
2/46
Michael Bolton
Tester, consultant, and trainer Michael Bolton is the coauthor (with James Bach) of Rapid SoftwareTesting, a course that presents a methodology and mindset for testing software expertly in uncertainconditions and under extreme time pressure. Michael is a leader in the context-driven software testing
movement with twenty years of experience testing, developing, managing, and writing about software.Currently, he leads DevelopSense, a Toronto-based consultancy. Prior to DevelopSense, he was withQuarterdeck Corporation, where he managed the companys flagship products and directed project andtesting teams--both in-house and worldwide. Contact Michael at [email protected].
-
8/11/2019 Rapid Testing Presentation
3/46
ARapidIntroduction
toRapidSoftwareTesting
JamesBach,Satisfice,Inc.
www.satisfice.com
+1(360)4401435
MichaelBolton,DevelopSense
www.developsense.com
+1(416)6565160
2
Acknowledgements Someofthismaterialwasdevelopedincollaborationwith
Dr.Cem Kaner,oftheFloridaInstituteofTechnology.Seewww.kaner.comandwww.testingeducation.org.
DougHoffman(www.softwarequalitymethods.com)hasalsocontributedtoandoccasionallyteachesfromthismaterial.
ManyoftheideasinthispresentationwerealsoinspiredbyoraugmentedbyothercolleaguesincludingJonathanBach,BretPettichord,BrianMarick,DaveGelperin,ElisabethHendrickson,JerryWeinberg,NoelNyman,andMaryAlton.
SomeofourexerciseswereintroducedtousbyPaysonHall,Jerry
Weinberg,Ross
Collard,
James
Lyndsay,
Dave
Smith,
Earl
Everett,
BrianMarick,Cem Kaner andJoeMcMahon.
Manyideaswereimprovedbystudentswhotookearlierversionsoftheclassgoingbackto1996.
-
8/11/2019 Rapid Testing Presentation
4/46
3
AssumptionsAboutYou
Youtest
software,
or
any
other
complex
human
creation.
Youhaveatleastsomecontroloverthedesignofyour
testsandsometimetocreatenewtests.
Youareworriedthatyourtestprocessisspendingtoo
muchtimeandresourcesonthingsthatarent
important.
Youtestunderuncertaintyandtimepressure.
Yourmajor
goal
is
to
find
important
problems
quickly.
Youwanttogetverygoodat(software)testing.
A
Question
What makes
testing harder or
slower?
-
8/11/2019 Rapid Testing Presentation
5/46
PremisesofRapidTesting
1. Softwareprojectsandproductsarerelationships
betweenpeople.
2. Eachprojectoccursunderconditionsofuncertainty
andtimepressure.
3. Despiteourbesthopesandintentions,some
degreeofinexperience,carelessness,and
incompetenceisnormal.
4. Atestisanactivity;itisperformance,notartifacts.
Premises
of
Rapid
Testing5. Testingspurposeistodiscoverthestatusofthe
productandanythreatstoitsvalue,sothatour
clientscanmakeinformeddecisionsaboutit.
6. Wecommittoperformingcredible,costeffective
testing,andwewillinformourclientsofanythingthat
threatensthatcommitment.
7. Wewillnotknowinglyornegligentlymisleadour
clientsand
colleagues
or
ourselves.
8. Testersacceptresponsibilityforthequalityoftheir
work,althoughtheycannotcontrolthequalityofthe
product.
-
8/11/2019 Rapid Testing Presentation
6/46
7
RapidTesting
Rapidtestingisamindset
andaskillsetoftesting
focusedonhowtodotesting
morequickly,
lessexpensively,
withexcellentresults.
This
is
a
general
testingmethodology.Itadaptsto
anykindofprojectorproduct.
8
Rapidtestingmaynot
beexhaustive,butitis
thoroughenoughand
quickenough.Itsless
workthanponderous
testing.Itmightbeless
workthan
slapdash
testing.
Itfulfillsthemission
oftesting.
HowdoesRapidTestingcompare
withother
kinds
of
testing?
Managementlikestotalk
aboutexhaustivetesting,but
theydontwanttofunditand
theydontknowhowtodoit.
Youcanalwaystest
quickly...
Butitmightbepoor
testing.
Whentestingisturnedintoan
elaboratesetofrotetasks,
itbecomesponderouswithout
reallybeingthorough.
M
oreWork&Time
(Cost)
BetterThinking&BetterTesting
(Value)
Slapdash
Muchfaster,
cheaper,
andeasier
PonderousSlow,expensive,
andeasier
RapidFaster,lessexpensive,
stillchallenging
ExhaustiveSlow,veryexpensive,
anddifficult
-
8/11/2019 Rapid Testing Presentation
7/46
9
ExcellentRapidTechnicalWork
BeginswithYou
When the ball comes to you
Doyouknowyouhavetheball?
Canyoureceivethepass?
Doyouknowwhatyour
roleandmissionis?
Doyouknowwhere
yourteammatesare?
Areyoureadytoact,rightnow?
Canyouletyourteammateshelpyou?
Doyouknowyouroptions?
Isyourequipmentready?
Canyoureadthe
situationonthefield?
Are
you
aware
of
the
criticalityofthesituation?
10
Rapidtestteamsareaboutdiversetalentscooperating
Wecallthistheellipticalteam,asopposedtotheteamof
perfectcircles.
Someimportantdimensionstovary:
Technicalskill
Domainexpertise
Temperament(e.g.introvertvs.extrovert)
Testingexperience
Projectexperience
Industry
experience Productknowledge
Educationalbackground
Writingskill
Diversitymakesexplorationfarmorepowerful
Yourteamispowerfulbecauseofyouruniquecontribution
butyoudonthavetobegreatateverything.
-
8/11/2019 Rapid Testing Presentation
8/46
WhatItMeansToTestRapidly
Sincetesting
is
about
finding
apotentially
infinite
numberofproblemsinaninfinitespaceinafinite
amountoftime,wemust
understandourmissionandobstaclestofulfillingit
knowhowtorecognizeproblemsquickly
modeltheproductandthetestspacetoknowwheretolook
forproblems
preferinexpensive,lightweight,effectivetools
reducedependenceonexpensive,timeconsumingartifacts,
whilegetting
value
from
the
ones
weve
got
donothingthatwastestimeoreffort
tellacrediblestoryaboutallthat11
One
Big
Problem
in
Testing
Formality Bloat
Much of the time, your testing doesnt need to be very formal*
Evenwhenyourtestingdoesneedtobeformal,youllneedto
dosubstantialamountsofinformaltestinginorderfigureout
howtodoexcellentformaltesting.
Whosays? TheFDA. Seehttp://www.satisfice.com/blog/archives/602
Even in a highly regulated environment, you do formal testing
primarily for the auditors. You do informal testing to make sure
you dont lose money, blow things up, or kill people.
*Formaltestingmeanstestingthatmustbedonetoverifyaspecificfact,
orthatmustbedoneinaspecificway.
-
8/11/2019 Rapid Testing Presentation
9/46
EXERCISE
Test the Famous Triangle
14
Whatistesting?Serving Your Client
Ifyoudonthaveanunderstandingandanagreement
onwhatisthemissionofyourtesting,thendoingit
rapidlywouldbepointless.
-
8/11/2019 Rapid Testing Presentation
10/46
NotEnough
ProductandProjectInformation?
Where do we get
test information?
What
Is
A
Problem?
A problem is
-
8/11/2019 Rapid Testing Presentation
11/46
HowDoWeRecognizeProblems?
An oracle is
a way to recognize
a problem.
Learn
About
HeuristicsHeuristicsarefallible,fastandfrugalmethodsofsolving
problems,makingdecisions,oraccomplishingtasks.
Theengineeringmethodis
theuseofheuristics
tocausethebestchange
in
a
poorly
understood
situation
withintheavailableresources.
BillyVaughanKoen
DiscussionoftheMethod
-
8/11/2019 Rapid Testing Presentation
12/46
Heuristics: GeneratingSolutions
QuicklyandInexpensively
Heuristic(adjective):
servingtodiscoverorlearn
Heuristic(noun):
afalliblemethodforsolvingaproblem
ormakingadecision
Heuristicreasoningisnotregardedasfinalandstrict
butas
provisional
and
plausible
only,
whose
purpose
istodiscoverthesolutiontothepresentproblem.GeorgePolya,HowtoSolveIt
OraclesAnoracleisaheuristic principleormechanism
bywhichwerecognizeaproblem.
...itappearedatleastoncetomeetsome
requirementto
some
degree
...uh,whenIranit
...thatonetime
...onmymachine.
Itworks!
really means
-
8/11/2019 Rapid Testing Presentation
13/46
FamiliarProblems
Ifaproductisconsistentwithproblemsweveseenbefore,
wesuspectthattheremightbeaproblem.
Explainability
Ifaproductisinconsistentwithourabilitytoexplainit
(orsomeoneelses),wesuspectthattheremightbeaproblem.
-
8/11/2019 Rapid Testing Presentation
14/46
World
Ifaproductisinconsistentwiththewaytheworldworks,
wesuspectthattheremightbeaproblem.
History
Ifaproductisinconsistentwithpreviousversionsofitself,
wesuspectthattheremightbeaproblem.
Okay,
so how the#&@ do I prin t
now?
-
8/11/2019 Rapid Testing Presentation
15/46
Image
Ifaproductisinconsistentwithanimagethat
thecompanywantstoproject,wesuspectaproblem.
Comparable
Products
WordPad Word
Whenaproductseemsinconsistentwithaproductthatis
insomewaycomparable,wesuspectthattheremightbeaproblem.
-
8/11/2019 Rapid Testing Presentation
16/46
Claims
Whenaproductisinconsistentwithclaimsthatimportant
peoplemakeaboutit,wesuspectaproblem.
User
Expectations
Whenaproductisinconsistentwithexpectationsthata
reasonableusermighthave,wesuspectaproblem.
-
8/11/2019 Rapid Testing Presentation
17/46
Purpose
When a product is inconsistent with its designers explicit
or impl icit purposes, we suspect a problem.
Product
When a product is inconsistent internallyas when it
contradicts itselfwe suspect a problem.
-
8/11/2019 Rapid Testing Presentation
18/46
StatutesandStandards
Whenaproductisinconsistentwithlawsorwidely
acceptedstandards,wesuspectaproblem.
32
Consistency(thisagreeswiththat)an
important
theme
in
oracle
principles
Familiarity:Thesystemisnotconsistentwiththepatternofanyfamiliarproblem.
Explainability: Thesystemisconsistentwithourabilitytodescribeitclearly.
World: Thesystemisconsistentwiththingsthatwerecognizeintheworld.
History: Thepresentversionofthesystemisconsistentwithpastversionsofit.
Image:Thesystemisconsistentwithanimagethattheorganizationwantstoproject.
ComparableProducts:Thesystemisconsistentwithcomparablesystems.
Claims: Thesystemisconsistentwithwhatimportantpeoplesayitssupposedtobe.
UsersExpectations:Thesystemisconsistentwithwhatuserswant.
Product:Eachelementofthesystemisconsistentwithcomparableelementsinthe
samesystem.
Purpose: Thesystemisconsistentwithitspurposes,bothexplicitandimplicit.
StandardsandStatutes:Thesystemisconsistentwithapplicablelaws,orrelevant
implicitorexplicitstandards.
Consistencyheuristicsrelyonthequalityofyour
modelsoftheproductanditscontext.
-
8/11/2019 Rapid Testing Presentation
19/46
Consistencyheuristicsrelyonthequalityof
yourmodelsoftheproductanditscontext.
AnoracledoesnttellyouthatthereISaproblem.
Anoracletellsyouthatyoumightbeseeingaproblem.
Relysolelyondocumented,anticipatedsourcesoforacles,
andyourtestingwilllikelybeslowerandweaker.
Trainyourmindtorecognizepatternsoforacles
andyourtestingwilllikelybefaster
andyourabilitytospotproblemswillbesharper.
AllOraclesAreHeuristic
Anoraclecanalertyoutoapossibleproblem,
butanoraclecannottellyouthatthereisnoproblem.
A person whose opinion matters.
An opinion held by a person who matters.
A disagreement among people who matter.
A reference document with useful information.
A known good example output.
A known bad example output.
A process or tool by which the output is checked.
A process or tool that helps a tester identify patterns.
A feeling like confusion or annoyance.
A desirable consistency between related things.
GeneralExamplesofOraclesthings that suggest problem or no problem
34
People
Mechanisms
Feelings
Principles
-
8/11/2019 Rapid Testing Presentation
20/46
Tacit Explicit
Other
People
Tester
Your
Feelings
Mental Models
Shared Artifacts
(specs, tools, etc.)
Stakeholders
Feelings
Mental Models
Inference
Observable
Consistencies
Reference
onference
Experience
Oracles from the Inside Out
Oracle
Cost
and
Value Someoraclesaremoreauthoritative
butmoreresponsivetochange
Someoraclesaremoreconsistent butmaybenotuptodate
Someoraclesaremoreimmediate butlessreliable
Someoraclesaremoreprecise buttheprecisionmaybemisleading
Someoraclesaremoreaccurate
butless
precise
Someoraclesaremoreavailable butlessauthoritative
Someoraclesareeasiertointerpret butmorenarrowlyfocused
-
8/11/2019 Rapid Testing Presentation
21/46
FeelingsAsHeuristicTriggersForOracles
Anemotional
reaction
is
atrigger
to
attention
andlearning
Withoutemotion,wedontreasonwell
SeeDamasio,TheFeelingofWhatHappens
Whenyoufindyourselfmildlyconcernedabout
something,someoneelsecouldbevery
concernedaboutit
Observeemotionstohelpovercomeyour
biasesand
to
evaluate
significance
An emotion is a signal; consider looking into it
38
All
Oracles
Are
Heuristic Weoftendonothaveoraclesthatestablishadefinitecorrectorincorrect
result,inadvance.Oraclesmayrevealthemselvestousonthefly,orlater.
Thatswhyweuseabductive inference.
Nosingleoraclecantelluswhetheraprogram(orafeature)isworking
correctlyatalltimesandinallcircumstances.
Thatswhyweuseavarietyoforacles.
Anyprogramthatlookslikeitsworking,toyou,mayinfactbefailingin
somewaythathappenstofoolallofyouroracles.Thatswhyweproceed
withhumilityandcriticalthinking.
Wenever
know
when
atest
is
finished.
Thatswhywetrytomaintainuncertaintywheneveryoneelseonthe
projectissure.
You(thetester)cantknowthedeeptruthaboutanyresult.
Thatswhywereportwhateverseemslikelytobeabug.
-
8/11/2019 Rapid Testing Presentation
22/462
39
OraclesareNotPerfect
AndTestersareNotJudges
Youdontneedtoknowforsureifsomethingisabug;
itsnotyourjobtodecideifsomethingisabug;itsyour
jobtodecideifitsworthreporting.
YoudoneedtoformajustifiedbeliefthatitMIGHTbe
athreattoproductvalueintheopinionofsomeone
whomatters.
Andyoumustbeabletosaywhyyouthinkso;youmust
beabletocitegoodoraclesoryouwilllosecredibility.
MIPing VS.BlackFlagging
40
CopingWith
Difficult
Oracle
Problems
IgnoretheProblem Asksowhat?Maybethevalueoftheinformationdoesntjustifythecost.
SimplifytheProblem Askfortestability.Itusuallydoesnthappenbyaccident. Builtinoracle.Internalerrordetectionandhandling. Lowerthestandards. Youmaybeusinganunreasonablestandardofcorrectness.
ShifttheProblem Paralleltesting. Comparewithanotherinstanceofacomparablealgorithm. Liveoracle.Findanexpertwhocantelliftheoutputiscorrect. Reversethefunction. (e.g.2x2=4,then4/2=2)
DivideandConquertheProblem Spotcheck.Performadetailedinspectionononeinstanceoutofasetofoutputs. Blinktest.Compareorreviewoverwhelmingbatchesofdataforpatternsthatstand
out.
Easyinput. Useinputforwhichtheoutputiseasytoanalyze. Easyoutput. Someoutputmaybeobviouslywrong,regardlessofinput. Unittestfirst.Learnaboutthepiecesthatmakethewhole. Testincrementally.Learnabouttheproductbytestingoveraperiodoftime.
-
8/11/2019 Rapid Testing Presentation
23/462
41
EasyInput
FixedMarkers.
Use
distinctive
fixed
input
patterns
that
are
easy
to
spotintheoutput.
StatisticalMarkers.Usepopulationsofdatathathavedistinguishablestatisticalproperties.
SelfReferentialData.Usedatathatembedsmetadataaboutitself.(e.g.counterstrings)
EasyInputRegions.Forspecificinputs,thecorrectoutputmaybeeasytocalculate.
OutrageousValues.Forsomeinputs,weexpecterrorhandling.
IdempotentInput.Tryacasewheretheoutputwillbethesameastheinput.
Match.Dothesamethingtwiceandlookforamatch.
ProgressiveMismatch.Doprogressivelydifferingthingsovertimeandaccountforeachdifference.(codebreakingtechnique)
42
OraclesAreLinkedToThreats
ToQuality
Criteria
Anyinconsistencymayrepresentdiminishedvalue.
Manytestapproachesfocusoncapability(functionality)
andunderemphasizetheothercriteria.
Capability Scalability
Reliability Compatibility
Usability Performance
Charisma Installability
Security Development
-
8/11/2019 Rapid Testing Presentation
24/462
43
OraclesAreLinkedToThreats
ToQualityCriteria
Anyinconsistencymayrepresentdiminishedvalue.
Manytestapproachesfocusoncapability(functionality)
andunderemphasizetheothercriteria.
Supportability
Testability
Maintainability
Portability
Localization
FocusingonPreparationandSkill
CanReduce
Documentation
Bloat
3.0TestProcedures
3.1Generaltestingprotocol. Inthetestdescriptionsthatfollow,thewordverify"isusedtohighlightspecificitems
thatmustbechecked.Inadditiontothoseitemsatestershall,atalltimes,bealertfor
anyunexplainedorerroneousbehavioroftheproduct.Thetestershallbearinmind
that,regardlessofanyspecificrequirementsforanyspecifictest,thereisthe
overarchinggeneralrequirementthattheproductshallnotposeanunacceptablerisk
ofharmtothepatient,includinganunacceptableriskusingreasonablyforeseeable
misuse.
Testpersonnelrequirements: Thetestershallbethoroughlyfamiliarwiththe
generator
and
workstation
FRS,
as
well
as
with
the
working
principles
of
the
devices
themselves.Thetestershallalsoknowtheworkingprinciplesofthepowertestjigandassociatedsoftware,includinghowtoconfigureandcalibrateitandhowtorecognizeif
itisnotworkingcorrectly.Thetestershallhavesufficientskillindataanalysisand
measurementtheorytomakesenseofstatisticaltestresults.Thetestershallbe
sufficientlyfamiliarwithtestdesigntocomplementthisprotocolwithexploratory
testing,intheeventthatanomaliesappearthatrequireinvestigation.Thetestershall
knowhowtokeeptestrecordstocredible,professionalstandard.
-
8/11/2019 Rapid Testing Presentation
25/462
Remember
Forskilledtesters,
goodtestingisntjustabout
passvs.fail.
Forskilledtesters,
testingisabout
problemvs.noproblem.
Where
Do
We
Look
For
Problems?
Coverage is
how much of the
product has been tested.
-
8/11/2019 Rapid Testing Presentation
26/462
Coverageishowmuchoftheproductwehavetested.
WhatISCoverage?
Itstheextenttowhichwehave
traveledoversomemapoftheproduct.
MODELS
Models Amodelisanidea,activity,orobject
suchasanideainyourmind,adiagram,alistofwords,aspreadsheet,
aperson,atoy,anequation,ademonstration,oraprogram
suchassomethingcomplexthatyouneedtoworkwithorstudy
Amapisamodelthathelpstonavigateacrossaterrain.
2+2=4isamodelforaddingtwoapplestoabasketthatalreadyhastwoapples.
Atmosphericmodelshelppredictwherehurricaneswillgo.
Afashionmodelhelpsunderstandhowclothingwouldlookonactualhumans.
Yourbeliefsaboutwhatyoutestareamodelofwhatyoutest.
thatheuristicallyrepresents(literally,
represents)anotheridea,activity,orobject
wherebyunderstandingsomethingaboutthe
modelmayhelpyoutounderstandormanipulate
thething
that
it
represents.
-
8/11/2019 Rapid Testing Presentation
27/462
Thereareasmanykindsoftestcoverageasthereare
waystomodelthesystem.
Intentionally OR Incidentally
OneWaytoModelCoverage:
ProductElements
(with
Quality
Criteria)
Capability
Reliability
Usability
Charisma
Security
Scalability
Compatibility
Performance
Installability
Supportability
Testability
Maintainability
Structure
Function
Data
Interfaces
Platform
Operations
Time
-
8/11/2019 Rapid Testing Presentation
28/462
51
Totestaverysimpleproductmeticulously,
partofacomplexproductmeticulously,
ortomaximizetestintegrity
1. Startthetestfromaknown(clean)state.
2. Prefersimple,deterministicactions.
3. Traceteststepstoaspecifiedmodel.
4. Followestablishedandconsistentlabprocedures.
5. Makespecificpredictions,observationsandrecords.
6. Makeiteasyto reproduce(automationmayhelp).
GeneralFocusingHeuristics
usetestfirstapproachorunittestingforbettercode
coverage
workfrompreparedtestcoverageoutlinesandrisklists
usediagrams,statemodels,andthelike,andcoverthem
applyspecifictesttechniquestoaddressparticularcoverage
areas
makecarefulobservationsandmatchtoexpectations
Todothismorerapidly,makepreparationandartifactsfastandfrugal:
leverageexistingmaterialsandavoidrepeatingyourself.
Emphasizedoing;relaxplanning. Youllmakediscoveriesalongtheway!
-
8/11/2019 Rapid Testing Presentation
29/462
53
Tofindunexpected problems,
elusive problemsthatoccurinsustainedfielduse,
ormoreproblemsquicklyinacomplexproduct
1. Start from different states (not necessarily clean).
2. Prefer complex, challenging actions.
3. Generate tests from a variety of models.
4. Question your lab procedures and tools.
5. Try to see everything with open expectations.6. Make the test hard to pass, instead of easy to reproduce.
Thats a
PowerPoint
bug!
GeneralDefocusingHeuristics
diversifyyourmodels;intentionalcoverageinoneareacanlead
tounintentionalcoverageinotherareasthisisaGoodThing
diversifyyourtesttechniques
bealerttoproblemsotherthantheonesthatyoureactively
lookingfor
welcomeandembraceproductivedistraction
dosometestingthatisnotorientedtowardsaspecificrisk
usehigh
volume,
randomized
automated
tests
-
8/11/2019 Rapid Testing Presentation
30/46
-
8/11/2019 Rapid Testing Presentation
31/462
ExtentofCoverage
Smokeand
sanity
Canthisthingevenbetestedatall?
Common,core,andcritical
Canthisthingdothethingsitmustdo?
Doesithandlehappypathsandregularinput?
Canitwork?
Complex,harsh,extremeandexceptional
Willthisthinghandlechallengingtests,complexdataflows,
andmalformedinput,etc.?
Willitwork?
HowMightWeOrganize,
Record,and
Report
Coverage?
automatedtools(e.g.profilers,coveragetools)
annotateddiagramsandmindmaps
coveragematrices
bugtaxonomies
MichaelHuntersYouAreNotDoneYetlist
JamesBachsHeuristicTestStrategyModel
describedatwww.satisfice.com
articlesaboutitatwww.developsense.com
MikeKellysMCOASTERmodel
productcoverage
outlines
and
risk
lists
sessionbasedtestmanagement
http://www.satisfice.com/sbtm
Seethreearticleshere:
http://www.developsense.com/publications.html#coverage
-
8/11/2019 Rapid Testing Presentation
32/46
-
8/11/2019 Rapid Testing Presentation
33/463
VisualizingTestProgress
VisualizingTestProgress
-
8/11/2019 Rapid Testing Presentation
34/463
VisualizingTestProgress
See A Sticky Situation , Better Software, February 2012
What
IS
Exploratory
Testing?
Simultaneous test design, test
execution, and learning.
James Bach, 1995
But maybe it would be a good idea to underscore
why thats important
-
8/11/2019 Rapid Testing Presentation
35/46
-
8/11/2019 Rapid Testing Presentation
36/463
WhyExploratoryApproaches?
Systemsare
too
complexfor
individualsto
comprehendand
describe
Productsevolve
rapidlyinways
thatcannotbe
anticipated
Inthefuture,developerswilllikelydomoreverificationandvalidationatthe
unitlevelthantheyhavedonebefore.
Testersmustexplore,discover,investigate,andlearnaboutthesystem.
Why
Exploratory
Approaches? Developersareusingtoolsandframeworksthat
makeprogrammingmoreproductive,butthat
maymanifestmoreemergentbehaviour.
Developersareincreasinglyadoptingunittesting
andtestdrivendevelopment.
Thetraditionalfocusisonverification,validation,
andconfirmation.
Thenewfocusmustbeonexploration,discovery,
investigation,andlearning.
-
8/11/2019 Rapid Testing Presentation
37/463
WhyExploratoryApproaches?
Wedont
have
time
to
waste
preparingwastefullyelaboratewrittenplans
forcomplexproducts
builtfrommanyparts
andinteractingwithmanysystems
(manyofwhichwedontcontrol
orevenunderstand)
whereeverything
is
changing
over
time
andtheressomuchlearningtobedone
andtheresult,nottheplan,isparamount.
QuestionsAbout
Scriptsarrows and cycles
Where do scripts
come from?
What happens when the
unexpected happens
during a script?
What do we do
with what we
learn?Will everyone fol low the same
script the same way?
(task performing)
-
8/11/2019 Rapid Testing Presentation
38/463
QuestionsAboutExplorationarrows and cycles
(value seeking)
Where does
exploration
come from?
What happens when
the unexpected
happens during
exploration?
What do we do
with what we
learn?
Will everyone
explore the same way?
Explorationis
Not
Just
Actionarrows and cycles
-
8/11/2019 Rapid Testing Presentation
39/463
Youcanputthemtogether!arrows and cycles
Youcanputthemtogether!arrows and cycles
-
8/11/2019 Rapid Testing Presentation
40/463
WhatExploratoryTestingIsNot
Touring http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart1touring/
AfterEverythingElseTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart2after
everythingelsetesting/
ToolFreeTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart3tool
freetesting/
QuickTests http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart4quick
tests/
UndocumentedTesting http://www.developsense.com/blog/2011/12/whatexploratorytestingisnotpart5
undocumented
testing/ ExperiencedBasedTesting
http://www.satisfice.com/blog/archives/664
definedbyanyspecificexampleofexploratorytesting http://www.satisfice.com/blog/archives/678
Exploratory
Testing
ISNOTrandomtesting(orsloppy,
orslapdashtesting)
ISNOTunstructuredtesting
ISNOTprocedurallystructured
ISNOTunteachable
ISNOTunmanageable
ISNOTscripted
ISNOTatechnique
ISadhoc,inthedictionarysense,
tothepurpose
ISstructuredandrigorous
IScognitivelystructured
IShighlyteachable
IShighlymanageable
ISchartered
ISanapproach
Thewaywepracticeandteachit,exploratorytesting
-
8/11/2019 Rapid Testing Presentation
41/463
ContrastingApproaches
ScriptedTesting Isdirectedfromelsewhere
Isdeterminedinadvance
Isaboutconfirmation
Isaboutcontrollingtests
Emphasizespredictability
Emphasizesdecidability
Likemakingaspeech
Likeplayingfromascore
ExploratoryTesting Isdirectedfromwithin
Isdeterminedinthemoment
Isaboutinvestigation
Isaboutimprovingtestdesign
Emphasizesadaptability
Emphasizeslearning
Likehavingaconversation
Likeplayinginajamsession
Exploratory
Testing
IS
Structured Exploratorytesting,asweteachit,isastructuredprocessconductedbya
skilledtester,orbylesserskilledtestersorusersworkingundersupervision.
ThestructureofETcomesfrommanysources: Testdesignheuristics
Chartering
Timeboxing
Perceivedproductrisks
Thenatureofspecifictests
Thestructureoftheproductbeingtested
Theprocess
of
learning
the
product
Developmentactivities
Constraintsandresourcesaffordedbytheproject
Theskills,talents,andinterestsofthetester
Theoverallmissionoftesting
Inotherwords,
itsnotrandom,
butsystematic.
Notprocedurally
structured,but
cognitivelystructured.
-
8/11/2019 Rapid Testing Presentation
42/464
79
Inexcellentexploratorytesting,onestructuretendsto
dominatealltheothers:
Exploratorytestersconstructacompellingstoryof
their
testing.
It
is
this
story
that
givesETabackbone.
ExploratoryTestingISStructured
80
Totestistocompose,edit,narrate,andjustify
THREEstories.
A story about the status of the PRODUCT
about how it failed, and how it might fail...
in ways that matter to your various clients.
A story about HOW YOU TESTED it
how you configured, operated and observed it
about what you havent tested, yet
and wont test, at all
A story about how GOOD that testing was
what the risks and costs of testing are
what made testing harder or slower
how testable (or not) the product is
what you need and what
you recommend.
-
8/11/2019 Rapid Testing Presentation
43/464
81
Whatdoestakingadvantageofresourcesmean?
Mission Theproblemweareheretosolveforourcustomer.
Information Informationabouttheproductorprojectthatisneededfortesting.
Developerrelations Howyougetalongwiththeprogrammers.
Team Anyonewhowillperformorsupporttesting.
Equipment&tools Hardware,software,ordocumentsrequiredtoadministertesting.
Schedule Thesequence,duration,andsynchronizationofprojectevents.
TestItems Theproducttobetested.
Deliverables Theobservableproductsofthetestproject.
82
Waystotest?
GeneralTest
Techniques
Functiontesting
Domaintesting
Stresstesting
Flowtesting
Scenariotesting
Claimstesting
Usertesting
Risktesting
Automaticchecking
-
8/11/2019 Rapid Testing Presentation
44/464
83
HappyPath
TourtheProduct
SampleData
Variables
Files
Complexity
Menus&Windows
Keyboard&Mouse
Aquicktestisacheaptestthathassomevalue
butrequireslittlepreparation,knowledge,
ortimetoperform.
Interruptions
Undermining
Adjustments
DogPiling
ContinuousUse
FeatureInteractions
ClickonHelp
CostasaSimplifyingFactor
Tryquicktestsaswellascarefultests
84
InputConstraintAttack
ClickFrenzy
ShoeTest
BlinkTest
ErrorMessageHangover
Aquicktestisacheaptestthathassomevalue
butrequireslittlepreparation,knowledge,
ortimetoperform.
ResourceStarvation
MultipleInstances
Crazy
Configs CheapTools
CostasaSimplifyingFactor
Tryquick
tests
as
well
as
careful
tests
-
8/11/2019 Rapid Testing Presentation
45/464
TouringtheProduct:
MikeKellysFCCCUTSVIDS
Featuretour
Complexitytour
Claimstour
Configurationtour
Usertour
Testabilitytour
Scenariotour
Variabilitytour
Interoperabilitytour
Datatour
Structuretour
-
8/11/2019 Rapid Testing Presentation
46/46
88
SummingUp:
Themesof
Rapid
Testing
Putthetester'smindatthecenteroftesting.
Learntodealwithcomplexity andambiguity.
Learntotellacompellingtestingstory.
Developtestingskillsthroughpractice,notjusttalk.
Useheuristicstoguideandstructureyourprocess.
Replacecheckforwithlookforproblemsin
Beaservicetotheprojectcommunity,notanobstacle.
Consider costvs.
valuein
all
your
testing
activity.
Diversifyyourteamandyourtactics.
Dynamicallymanagethefocusofyourwork.
Yourcontextshoulddriveyourchoices,bothofwhichevolveovertime.