planning your agile testing: a practical guide
DESCRIPTION
Traditional test plans are incompatible with agile software development because we don't know all the details about all the requirements up front. However, in an agile software release, you still must decide what types of testing activities will be required—and when you need to schedule them. Janet Gregory explains how to use the Agile Testing Quadrants, a model identifying the different purposes of testing, to help your team understand your testing needs as you plan the next release. Janet introduces you to alternative, lightweight test planning tools that allow you to plan and communicate your big picture testing needs and risks. Learn how to decide who does what testing—and when. Determine what types of testing to consider when planning an agile release, the infrastructure and environments needed for testing, what goes into an agile “test plan,” how to plan for acquiring test data, and lightweight approaches for documenting your tests and recording test results.TRANSCRIPT
1/22/2013
1
Janet Gregory, DragonFire Inc.Copyright 2013
Star Canada 2013
With material from Lisa Crispin
Copyright 2013 DragonFire Inc.
My experience comes …As a tester, working on agile teams
Coaching and training, learning
Agile Testing: A Practical Guide for Testers and Agile Teams; Addison Wesley 2009
1/22/2013
2
Copyright 2013 DragonFire Inc.
How many years experience with agile?◦ At least 2 years◦ Less than 2 years◦ Less than 1 year◦ Have done some reading, starting in a team◦ Completely new to the concept
Are you a ...◦ Tester◦ Test Manager◦ Programmer◦ Other
3
Copyright 2013 DragonFire Inc.
10 minutes ~1 minute / person
NameWhere you are from?Why you are here?
Let everyone have a chance to introduce themselves
4
1/22/2013
3
Copyright 2013 DragonFire Inc. 5
Copyright 2013 DragonFire Inc.
• have short iterations• encourage active customer participation• demand whole team collaboration• test features & stories as they are coded• deliver business value at regular intervals• adapt their processes based on feedback
6
1/22/2013
4
Sequential, phased – eg. waterfall
7
It 1 It 2 It 3
A B A BC
Time
Agile: iterative, incremental– Each story is expanded, coded and
tested– Possible release after each iteration
Requirements
SpecificationsCode
TestingRelease
It 0
A BCD
It 4
A BCD
FE
Copyright 2013 DragonFire Inc.
What concerns you about agile
and test planning?
1/22/2013
5
Copyright 2013 DragonFire Inc.
Test Approach – The Agile WayProject Initiation Get an understanding of the project (business)
Participate in sizing stories Create Test Plan Release/Project Planning
Each Iteration1 ….X
Estimate tasks, Run regression testsCollaborate with customers on acceptance tests
Write, automate and execute new story testsPair test with other testers, developers
Perform exploratory testing
Perform Final Load TestComplete Final Regression Test
Perform UATPerform Mock Deploy
Participate in Release Readiness
System Test / End Game
Participate in Release to ProdParticipate in Retrospectives
Release to Prod/ Support
9
Copyright 2013 DragonFire Inc.
Story Tests
Project Test Plan
10
Release Test Plan
1/22/2013
6
Copyright 2013 DragonFire Inc.
• Product Roadmap◦ High level feature ideas
• Release Planning◦ Backlog of sized and prioritized stories
• Iteration Planning◦ Backlog of estimated tasks◦ Estimations are in hours: how long will it take
11
Copyright 2013 DragonFire Inc.
• Types of testing needed• Automation• Complexity• Size
More details later …
12
1/22/2013
7
Copyright 2013 DragonFire Inc.
Instead of saying NO, or being the gatekeeper
Be the information provider so business can make the decisions
Questions… on agile approach to planning
13
Copyright 2013 DragonFire Inc. 14
1/22/2013
8
Copyright 2013 DragonFire Inc. 15
Let’s talk a little testing ….
What is enough to keep you out
of trouble?
Copyright 2013 DragonFire Inc.
• Find bugs• Check correctness• Test reliability• Check usability• Answer “Is it done?”• Learn about the application• Feedback into future stories
What else???
16
1/22/2013
9
Copyright 2013 DragonFire Inc.
• Each group, collaborate • Write different types of testing that you do• One type per sticky• 5 minutes
17
Copyright 2013 DragonFire Inc.
• Model for classifying tests• Looks at the purpose of the tests – the ‘why’
18
1/22/2013
10
19
Agile Testing Quadrants Brain child of Brian Marick
Copyright 2013 DragonFire Inc.
• Take your tests and put them in the quadrant you think they belong in.
• Were you able to get them all classified?
• Add tests as you think about them.
20
1/22/2013
11
Agile Testing Quadrants (Brian Marick)
21
Copyright 2013 DragonFire Inc.
• Can be used as a communication toolo to explain testing in a common language
• Emphasize whole-team responsibilityo focus on collaborationo whole team participation
• Help plan what tests to automateo and tools and infrastructure needed
1/22/2013
12
Copyright 2013 DragonFire Inc.
For each story / feature◦ No story is done until tested◦ Customer needs captured as passing tests◦ Automated regression tests
For release readiness◦ Delivers value◦ “Doneness” in all quadrants
23
Copyright 2013 DragonFire Inc. 24
1/22/2013
13
Copyright 2013 DragonFire Inc.
Unit Testso Tests developer intent - program designo Tests a small piece of code o Makes sure it does what it shouldo TDD
Component Testso Tests architecture intent – system designo Tests that components work together correctly
Connectivity Tests
Copyright 2013 DragonFire Inc.
• Builds quality in• Focus on internal code quality• Builds testability into code• Go faster, do more
o Unit tests provide refactoring support• Provides instant feedback • TDD increases confidence in design• Forms the foundation of automation suite
1/22/2013
14
27
Can you think
of anything else
for Quadrant 1?
Copyright 2013 DragonFire Inc. 28
1/22/2013
15
Copyright 2013 DragonFire Inc.
• Use to elicit requirements• Acceptance Test (or Example) Driven Development
o Allows developers to code until the tests pass• Capture examples, express as executable tests• User experience
o wire frames; mock-ups / prototypes
Copyright 2013 DragonFire Inc.
• Help customers achieve advance clarity• Gain shared common understanding of stories• Drive development with business-facing tests• Obtain enough requirements to start coding• Focus is external quality• Customer – developer – tester collaboration◦ Power of Three
1/22/2013
16
31
Can you think
of anything else
for Quadrant 3?
Discussions?
Copyright 2013 DragonFire Inc. 32
1/22/2013
17
Copyright 2013 DragonFire Inc.
• Exploratory testingo what didn’t we think about
• Test for usabilityo understand end users, personas
• Tours• User Acceptance Testing (UAT)
Copyright 2013 DragonFire Inc.
• Iteration reviews / demoso Builds confidenceo Quick feedback loop
• Pair test with customers• Informal demos
o Pair exploratory testing with customer
1/22/2013
18
Copyright 2013 DragonFire Inc.
• Provide feedback ….o Turn learnings into tests that drive new featureso Change process as needed
• Evaluation of the product• Recreate actual user experiences• Realistic use
Copyright 2013 DragonFire Inc.
• Know your customers• Make them real• Plan your exploratory
testing using them
• Picture – from Jeff Patton’s Pragmatic Personas weekly column on Sticky Minds (1/25/2010)
1/22/2013
19
Copyright 2013 DragonFire Inc.
• Think of some personas or characters and devise exploratory scenarios which that role might get into, for example:
Shopping website - Amazono Senior citizen who never shopped on the webo Hacker looking to cause troubleo Working mom in a rusho Internet-savvy teen
37 37
38
Any questions about
Quadrant 3?
1/22/2013
20
Copyright 2013 DragonFire Inc. 39
Copyright 2013 DragonFire Inc.
• Non-functional tests• Performance, scalability, stress, load• “ility” testing• Automated deployments• Memory management• Infrastructure testing• Security testing
o Roles & permissions , system ‘hacking’• Data migration• Recovery
1/22/2013
21
• May be higher priority than functional req’ts• Makes the ‘finished’ product• Transfer specialized expertise
Copyright 2013 DragonFire Inc.
Consider the four quadrants • Would you add anything?• Consider your quality attributes
• Do you need to move any tests?• What tests are you missing?• What is your team is not doing now?
42
1/22/2013
22
push the tests lower
Automate at the feature
level
Automate at the story
level
Automate at the task
level
Sequential, phased – eg. waterfall
44
It 1 It 2 It 3
A B A BC
Time
Agile: iterative, incremental– Each story is expanded, coded and
tested– Possible release after each iteration
Requirements
SpecificationsCode
TestingRelease
It 0
A BCD
It 4
A BCD
FE
All automation done here
Automation hereAutomation
here
Automation here
Automation here
1/22/2013
23
Copyright 2013 DragonFire Inc.
Instead of saying NO, or being the gatekeeper
Be the information provider so business can make the decisions
Questions… on types of testing?
45
Copyright 2013 DragonFire Inc. 46
1/22/2013
24
Copyright 2013 DragonFire Inc. 47
Release Level Test Planning
Copyright 2013 DragonFire Inc.
• Consider scope, priorities, risks• Document only what is useful• Consider all types of testing• Budget time for:
o infrastructureo toolso automation needs
• Strive for simplicity
48
1/22/2013
25
49
Copyright 2013 DragonFire Inc.50
• Sizing of stories◦ think about impacts to system
• Ask questions ◦ that may impact the ‘bigness’ of the story
◦ to uncover hidden assumptions
◦ such as ‘what if?’, or “what happens if?”
• Not the time for details• Create a project “test plan”
1/22/2013
26
Copyright 2013 DragonFire Inc.
• Are we working with a vendor?◦ If so, how do we coordinate testing?
• Are there dependencies between … ◦ features?◦ teams?◦ Stories?
51
1/22/2013
27
Copyright 2013 DragonFire Inc.
• Should be project specific• High-light critical information◦ risks◦ assumptions◦ constraints
• Focus on what is really needed• Put static information –in a Quality
Management Strategy document
53
Copyright 2013 DragonFire Inc. 54
1/22/2013
28
Copyright 2013 DragonFire Inc. 55
Copyright 2013 DragonFire Inc.
A test matrix (one example)◦ provides a view of the release◦ can also be used as visible progress
But, remember◦ the value is in the planning
Let`s look at an example
56
1/22/2013
29
Copyright 2013 DragonFire Inc.
Copyright 2009 Janet Gregory, DragonFire 57
Copyright 2013 DragonFire Inc. 58
Let’s do some collaboration,
& work together to find a solution
1/22/2013
30
Copyright 2013 DragonFire Inc.
Sub topic
Sub topic
Sub topic
MAINTOPIC
Sub topic
Sub topic
first time
change
new account
passwordrules
encryption
save user name
rules
59
Copyright 2013 DragonFire Inc.
Mind map testing ideas for a feature
How many ideas can you generate in 5 minutes?
1/22/2013
31
Copyright 2013 DragonFire Inc.
Feature A• As an on-line shopper, I want to provide my
shipping address when I check out so that my order goes to the right location
Feature B• As an on-line shopper, I want to provide
payment information when I check out so I am billed correctly.
Copyright 2013 DragonFire Inc.
• Create a test matrix for the stories on the next slide.
• Functionality down the left side• Test conditions across the top
o Think heuristics o Use mnemonics (ex. SFDPOT)
• Gray out the squares that are not applicable.
1/22/2013
32
Copyright 2013 DragonFire Inc.
Feature A• As an on-line shopper, I want to provide my
shipping address when I check out so that my order goes to the right location
Feature B• As an on-line shopper, I want to provide
payment information when I check out so I am billed correctly.
Copyright 2013 DragonFire Inc.
• What value do you see in something like this?• What did you learn?• How would you use it?
1/22/2013
33
Copyright 2013 DragonFire Inc.
Instead of saying NO, or being the gatekeeper
Be the information provider so business can make the decisions
Questions on test planning at the release level?
65
Copyright 2013 DragonFire Inc. 66
1/22/2013
34
Copyright 2013 DragonFire Inc.
Iteration Planning: • Stories tell us about the tip of the iceberg• What don’t we know?• What questions should we ask to find out?
68
1/22/2013
35
Copyright 2013 DragonFire Inc. 69
Create a user story
Write Customer (Q2) Tests
Start thinking
how to code
Expand tests –
Story TestsAutomate Q2 Tests
Exploratorytesting
TDD
Customer User
Acceptance
Product ownerProduct owner/ TesterTesterTester/ProgrammerProgrammer
Pair,“Show Me”
Copyright 2013 DragonFire Inc.70
• Be proactive – preplanning • Try specifications workshops – Gojko Adzic
• Help customer achieve ‘advance clarity’ on stories◦ Customers speak with “one voice”◦ Testable stories◦ Steel threads◦ Create acceptance tests
1/22/2013
36
Copyright 2013 DragonFire Inc. 71
• Define high level story tests or examples • Find hidden assumptions• Define and estimate testing tasks – consider:
o automation needso test data, o exploratory testingo Q4 tests (‘ilities’, security, performance, etc.)
Copyright 2013 DragonFire Inc.
Ask questions• What's the business goal?• Can the user mess up?• What’s the best thing?• What’s the worst thing that can happen?• Watch for scope creep or ‘bling’• Is this story testable?
72
1/22/2013
37
Copyright 2013 DragonFire Inc. 73
Picture from Mike Cohn’s website
73
To Review
Copyright 2013 DragonFire Inc.
• Express the intent of the story• Use examples as specific instances of a scenario• Think expected and unexpected behaviors• Gives a shared common understanding of the
story• Feeds into TDD (Test Driven Development)
74
1/22/2013
38
Copyright 2013 DragonFire Inc.
As a new user, I want to create an account with a user name and
password so that only I can access my information.
===============================
75 75
Copyright 2013 DragonFire Inc.
• To create my account, I enter a valid user name and password, my information is saved and I am logged into the system and on the home page.
• If I enter an invalid user name, I get an error message “Invalid User Name”, and I am able to try again.
• If I enter an invalid password, I get an error message “Invalid Password”, and I am able to try again.
76
1/22/2013
39
Copyright 2013 DragonFire Inc.
BDD – Behavioural Driven Development
Given the user has no existing accountWhen she requests to create a new account,Then she enters a valid user name and valid
password (rules defined) And the information is saved upon submitting .
77 77
Copyright 2013 DragonFire Inc. 78
returnValue TestLogin ( userName, password)
User Name Password Expected result comments
JanetGregory Password Access system as JanetGregory
Valid combo saved
Janet Gregory Password Error Space in user name
JanetGregory Abc Error Invalid password
1/22/2013
40
Copyright 2013 DragonFire Inc.
• Write Acceptance Test(s) for one of the two stories – any format you like
• To help, try mind-mapping or draw a flow diagram
• Ask your customer for examples.• Think “Amazon”
• (~15 min)
Copyright 2013 DragonFire Inc.
Story a – Part of Feature A• As the company shipper, I need to verify the
city, state and postal code so that the order goes to the right location.
Story b – Part of Feature B• As the company accountant, I need to ensure
the credit card information is correct so the on-line shopper is billed correctly.
1/22/2013
41
We walk out of the iteration planning meeting ....
Now what?
Copyright 2013 DragonFire Inc. 82
1/22/2013
42
Copyright 2013 DragonFire Inc.
• Start simple with the high level acceptance tests◦ Add boundary, edge conditions, etc.◦ Add non-functional tests
• Experiment to find the right detail level• Review with programmers◦ Then automate
• Keep adding until story is complete• Consider exploratory test scenarios
83
Copyright 2013 DragonFire Inc.
Acceptance Tests – Fit Automation Style-- Third thread
84
User Name Password Expected result Comments
JanetGregory Password Access system as JanetGregory
Valid combo saved
Janet Gregory Password Error Space in user name
Janet#Gregory Password Error Special char not allowed
Password Error Blank user name
JanetGregory Password Error User already exists
JanetGregory Abc Error Not enough char in password
JanetGregory Error Blank password
1/22/2013
43
Copyright 2013 DragonFire Inc.
How do you know you are done?
85
Copyright 2013 DragonFire Inc.
Description Completed
1 Story tests reviewed, automated and pass Yes
2 Exploratory testing complete Yes
3 Unit tests reviewed and pass Yes
4 Q4 tests complete Yes
5 Acceptance tests pass Yes
86
1/22/2013
44
Copyright 2013 DragonFire Inc.
Instead of saying NO, or being the gatekeeper
Be the information provider so business can make the decisions
Questions on test planning at the story level?
87
Copyright 2013 DragonFire Inc. 88
1/22/2013
45
Copyright 2013 DragonFire Inc.
• Understand your context• Understand the purpose• Consider ROI (return on investment)• Push the tests lower• Automate the repetitive and boring tests• Plan, but document simply• Plan, but plan for the appropriate level
Copyright 2013 DragonFire Inc.
• How much is too much?• What is not enough?• Why do you need them?• What is the right information?• Who looks at them?• What do they use them for?• What is the simplest way you can capture
them?
90
1/22/2013
46
Copyright 2013 DragonFire Inc.
• What problem are you trying to solve?• What measurements make sense?• What is the simplest way to gather them?• Who needs to see them?• What is the simplest way to make them visible?• How are you going to use them?• When will you know to stop?• Be Aware!!!
o The wrong measurements can be deadlyo Metrics can be misinterpreted
Copyright 2013 DragonFire Inc.
• How are you going to know when you are done?• What is enough?• Collaborate to decide what coverage you want• Think risk – include the customer• Functional coverage• Lines of code? … be careful• What tools can you use?
o Heuristicso Mnemonics
1/22/2013
47
Copyright 2013 DragonFire Inc.
• Understand the level of precision you need◦ Think big picture in release planning◦ Think tasks during iteration planning◦ Think tests during story test planning
• Use ATDD for feature and story testing• Make it visible• Make it simple• Make it valuable• You want to be testing, not documenting
Copyright 2013 DragonFire Inc.
Instead of saying NO, or being the gatekeeper
Be the information provider so business can make the decisions
Are there still unanswered questions.. Concerns?
94
1/22/2013
48
Copyright 2013 DragonFire Inc.
Now AvailableAgile Testing: A Practical Guide for Testers and Agile Teams
By Lisa Crispin and Janet Gregory
www.agiletester.ca
My contact infowww.janetgregory.cahttp://janetgregory.blogspot.com/Email: [email protected]://lisacrispin.comEmail: [email protected]
95
Copyright 2013 DragonFire Inc.
• www.lisacrispin.com • [email protected]• http://www.exampler.com - Brian Marick’s web site• www.testobsessed.com (heuristics cheat sheet) – Elisabeth Hendrickson• http://www.shino.de/blog/ - Markus Gärtner’s blog• Gojko Adzic, Bridging the Communication Gap, 2009• Gojko Adzic, Executable Specifications, Manning, 2011• vwww.mountaingoatsoftware.com – Mike Cohn’s web site (and all his
books)• http://www.developsense.com/2009/04/of-testing-tours-and-
dashboards.html • James Whittaker, Exploratory Software Testing, 2009 Addison-Wesley• http://www.stickyminds.com/ Pragmatic Personas - Jeff Patton’s weekly
column 1/25/2010• Jean Tabaka, Collaboration Explained, 2006 Addison-Wesley• Agile Manifesto: http://agilemanifesto.org/
96