cqaa exploratory testing in the enterprise 6.22.2011 2011/cqaa exploratory... · pure scripted...
TRANSCRIPT
![Page 1: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/1.jpg)
Exploratory Tes.ng in the Enterprise
CQAA June 22, 2011
![Page 2: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/2.jpg)
Partner, DeveloperTown www.DeveloperTown.com
Past President, Associa.on for SoFware Tes.ng www.Associa.onForSoFwareTes.ng.org
Ar.cles and Blogs www.MichaelDKelly.com
Author of the chapter on “Session Based Test Management” in upcoming book How to Reduce the Cost of So0ware Tes4ng
![Page 3: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/3.jpg)
A STORY
![Page 4: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/4.jpg)
• 3 testers instead of 15 • 66% (or $200,000) cheaper than budgeted • two-‐week shorter window for tes.ng • delivered on .me • no serious defects in produc.on
HOW?
![Page 5: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/5.jpg)
“Simultaneous learning, test design, and test execu.on.”
-‐ James Bach
![Page 6: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/6.jpg)
pure scripted freestyle exploratory
charters vague scripts
fragmentary test cases (scenarios) roles
-‐image from Jon Bach
![Page 7: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/7.jpg)
the tester, the way they work,
and their ability to work in the future, is as important as
the product being tested
![Page 8: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/8.jpg)
What makes them “exploratory?”
![Page 9: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/9.jpg)
Self Management
![Page 10: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/10.jpg)
Developing Ideas
![Page 11: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/11.jpg)
Examining the Product
![Page 12: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/12.jpg)
There’s always resistance
![Page 13: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/13.jpg)
A desire for visible, track-‐able, progress
![Page 14: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/14.jpg)
It’s difficult to quan.ta.vely measure coverage
![Page 15: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/15.jpg)
There’s extraordinary pressure for “reuse” and “cost control”
![Page 16: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/16.jpg)
Session Based Test Management
![Page 17: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/17.jpg)
• charters
![Page 18: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/18.jpg)
• charters • .me-‐boxed
![Page 19: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/19.jpg)
• charters • .me-‐boxed • session notes
![Page 20: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/20.jpg)
• charters • .me-‐boxed • session notes • debriefs
![Page 21: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/21.jpg)
• charters • .me-‐boxed • session notes • debriefs • team priori.za.on
![Page 22: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/22.jpg)
• charters • .me-‐boxed • session notes • debriefs • team priori.za.on • ad-‐hoc test documenta.on
![Page 23: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/23.jpg)
• charters • .me-‐boxed • session notes • debriefs • team priori.za.on • ad-‐hoc test documenta.on • ad-‐hoc test automa.on
![Page 24: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/24.jpg)
• charters • .me-‐boxed • session notes • debriefs • team priori.za.on • ad-‐hoc test documenta.on • ad-‐hoc test automa.on • dynamic metrics and repor.ng
![Page 25: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/25.jpg)
![Page 26: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/26.jpg)
Focus on dealing with uncertainty
![Page 27: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/27.jpg)
START
![Page 28: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/28.jpg)
Begin by implemen.ng a specific prac.ce
![Page 29: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/29.jpg)
Add exploratory tes.ng days to the end of exis.ng test cycles
![Page 30: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/30.jpg)
Carve out a small por.on of an exis.ng project and apply exploratory tes.ng to that por.on
![Page 31: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/31.jpg)
Run a pilot project
![Page 32: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/32.jpg)
Responding to concerns
![Page 33: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/33.jpg)
the Michael Jordan effect
![Page 34: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/34.jpg)
“We don’t do agile.”
![Page 35: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/35.jpg)
“What about regression tes.ng?”
![Page 36: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/36.jpg)
“We hand our test scripts off for test automa.on…”
![Page 37: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/37.jpg)
“How do you provide traceability?”
![Page 38: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/38.jpg)
“Your percent complete keeps changing!”
![Page 39: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/39.jpg)
“Don’t we lose the ability to add people late in
the project if we get sideways?”
![Page 40: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/40.jpg)
“Our testers aren’t subject mager experts.”
![Page 41: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/41.jpg)
“We can’t afford all senior testers.”
![Page 42: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/42.jpg)
“How do you make charters repeatable?”
![Page 43: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/43.jpg)
“What about es.ma.on?”
![Page 44: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/44.jpg)
Ques.ons?
![Page 45: CQAA Exploratory Testing in the Enterprise 6.22.2011 2011/CQAA Exploratory... · pure scripted freestyle exploratory vague scripts charters fragmentary test cases (scenarios) roles](https://reader030.vdocuments.mx/reader030/viewer/2022021415/5ad025397f8b9ad24f8d55aa/html5/thumbnails/45.jpg)
Reading Materials
• Bach, James. "Exploratory Tes.ng Explained." First published as a chapter in The Test Prac..oner, 2002. Available online at: hgp://www.sa.sfice.com/ar.cles/et-‐ar.cle.pdf
• Bach, James and Jon Bach. "Exploratory Tes.ng Dynamics." Available online at: hgp://www.sa.sfice.com/ar.cles/et-‐dynamics.pdf
• Bach, Jon. "Session Based Test Management." First published in SoFware Tes.ng and Quality Engineering, November 2000. Available online at: hgp://www.sa.sfice.com/ar.cles/sbtm.pdf
• Bach, James. "A Low-‐Tech Tes.ng Dashboard." Presented at STAREast, 1999. Available online at: hgp://www.sa.sfice.com/presenta.ons/dashboard.pdf
• Bach, Jon. "How to Manage and Measure Exploratory Tes.ng." Quardev, 2006. Available online at: hgp://www.quardev.com/content/whitepapers/how_measure_exploratory_tes.ng.pdf
• Pegy, Kenn. "Reflec.ons on the Use of Session-‐Based Exploratory Tes.ng As the Primary Test Methodology for SoFware in an Agile Environment." Presented at the Indianapolis Workshops on So0ware Tes4ng, April 2005. Available online at: hgp://www.indianapolisworkshops.com/docs/Reflec.ons_on_the_use_of_Session-‐Based_Exploratory_Tes.ng_in_an_Agile_Environment.doc
• Pegy, Kenn. "Transi.oning from Standard V&V to Rapid Tes.ng Prac.ces in a Chao.c Project Environment." Presented at the Conference of the Associa4on for So0ware Tes4ng, 2007. Available online at: hgp://www.associa.onforsoFwaretes.ng.org/documents/Kenn_Pegy_Transi.oning_from_Standard_V&V_to_Rapid_Tes.ng_Prac.ces.pdf
• Wood, Bill and David James. "Applying Session-‐Based Tes.ng to Medical SoFware." Medical Device and Diagnos.c Industry Magazine, May 2003. Available online at: hgp://www.mddionline.com/ar.cle/applying-‐session-‐based-‐tes.ng-‐medical-‐soFware
• Kelly, Michael. “The benefits of exploratory tes.ng in agile environments.” SearchSoFwareQuality.com, 2009. Available online at: hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1350741,00.html
• Kelly, Michael. “Using session-‐based test management for exploratory tes.ng.” SearchSoFwareQuality.com, 2009. Available online at: hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1352925_mem1,00.html
• Kelly, Michael. “Using session-‐based test management for exploratory tes.ng coverage problems.” SearchSoFwareQuality.com, 2009. Available online at: hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1355475_mem1,00.html
• Kelly, Michael. “Solving problems with session-‐based test management.” SearchSoFwareQuality.com, 2009. Available online at: hgp://searchsoFwarequality.techtarget.com/.p/0,289483,sid92_gci1355815_mem1,00.html