steven k allott - effective testing - softtest ireland
DESCRIPTION
Visit SoftTest Ireland www.softtest.ie and sign up for access to free Irish Software Testing events.TRANSCRIPT
Presented by Stephen Allott
CaseMaker InternationalA Diaz & Hilterscheid group company
Effective Testing
Using industry standard techniques
SoftTest Ireland 16th February 2006
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 2
Agenda
• Why do we test software?
• How do we test effectively?
• Risk based testing• Black Box techniques• Pairwise testing
Photo: Courtesy of NASA
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 3
Why test software?
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 4
Read all about it!
US site – down for 3 days,
forced to return $30 fee to nearly
260,000 online “fantasy
baseball” players.
US site down for 2
days, $280,000 in
lost sales.
Shopping Site: 6-hour downtime:
$400,000 in direct costs plus additional
in discounts to inconvenienced
customers.
UK - Concerns over bugs
and delays in new air
traffic control software
UK – Payment clearing system failure at month end
delayed wages to many customers
UK - Televisions,
Cameras sold at
very low prices
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 5
Ineffective Testing
• Creates too many of the wrong tests• Duplicates effort unnecessarily• Costs too much and takes too long• Has an adverse impact on the business as a
whole
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 6
Effective Testing
• Is aligned with the business need• Uses a risk-based approach and formal
techniques to create fewer tests with increased coverage
• Facilitates communication with the business• Provides pre-production confidence• Saves time and money
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 7
Risk Based Testing
• Identify business risks• Assess the impact
– Severity, if it were to fail– Probability of failure
• Focus more testing on high risk areas
• Continually measure and monitor the risks
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 8
Test Case Design Techniques – 1
• Black box and white box techniques have been proven to be effective
• Techniques are described in BS7925• Analysis/design separated from execution• Improved and demonstrable test coverage
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 9
Test Case Design Techniques – 2
• Analysis of the test conditions• A formal design process gives you
– Objectivity– Repeatability– Consistency– Documentation– Auditability– Traceability– Confidence
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 10
Test Case Design Techniques – 3
• The design activity alone can find faults• Helps drive the need for good requirements• Use of formal techniques helps with estimates
of overall test effort required• Supports test automation
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 11
Black Box Techniques
• BS7925 techniques– Equivalence class
partitioning– Boundary value
analysis– Decision tables– State transition
testing– Syntax testing– Random testing
• Other techniques– Pairwise– Domain analysis– Use case testing
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 12
White Box Techniques
B
S
MCDC
BC
L
All Paths
DU
DPU
DCU
where: B - Branch coverage/Decision coverage BC - Branch Condition coverage BCC - Branch Condition Combination coverage DCU - All C-uses coverage DPU - All P-uses coverage DU - All du-paths coverage L - LCSAJ coverage MCDC - Modified Condition Decision coverage S - Statement coverage
BCC
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 13
Test Data
• Copy of live data• Realistic data volumes• Range of values and
inputs
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 14
Equivalence Class Partitioning
• Can you ever see a need to: – try the view from every seat of a cinema or theatre?– test drive every car in the showroom?– withdraw cash from all the ATMs in your country?
• If you answered “No, of course not!”– what would you do instead, and why?
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 15
Equivalence Class Partitioning
• Equivalence class or partition• Ranges of data• Based on an assumption about the software’s
behaviour• Reduces the number of test cases whilst
maintaining reasonable test coverage
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 16
Equivalence Class Partitioning
• An online car insurance quotation system allows a 30% discount for good drivers aged between 25 and 50 but charges full price if you are under 25. For older careful drivers a 20% discount is allowed and if you are retired we will be happy to quote you a special price
Under 17 17 to 24 25 to 49 50 to 64 65 and over
£1,000 £700 £800
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 17
Boundary Value Analysis
• Builds on Equivalence Class technique• Assumes errors are made on the boundaries• Applicable to value and structure of data
– where there is a continuous range of values– length and character type
• Generates three test cases per boundary– one on the boundary– one either side of the boundary– the smallest distance away
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 18
Boundary Value Analysis – Example
• An online car insurance quotation system – Under 17 – unable to quote– 17 to 24 (premium price)– 25 to 49 (30% discount)– 50 to 64 (20% discount)– 65 and over (price on application)
Under 17 17 to 24 25 to 49 50 to 64 65 and over
16,17,18 49,50,5124,25,26 64,65,66
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 19
Boundary Value Analysis – Example
• Online Banking (Account balance)• Lower boundaries
– EUR 0.01 (DR)– EUR 0.00 (CR)– EUR 0.01 (CR)
• Upper boundaries– EUR 999,999.99 (CR)– EUR 1,000,000.00 (CR)– EUR 1,000,000.01 (exceeds max for this account
type)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 20
Boundary Value Analysis – Example
• Online Banking (Set up standing order)• Boundaries for from date
– Yesterday (invalid)– Today (should be allowed)– Tomorrow (should be allowed)
• Boundaries for until date– 12 months from today (ok)– 12 months less 1 day (ok)– 12 months and one day from today (invalid)
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 21
Decision Tables
• Another intuitive black box technique• Express complex business rules in a table• Cause-Effect graphing helps with the analysis • Define the combination of conditions (causes)• Define the resulting actions (effects)• Easy to create test cases
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 22
Example – Train Tickets
Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Holds a railcard Y Y N N
Travels after 9 am
Y N Y N
ActionsPay £10
Pay £15
Pay £20
Pay £30
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 23
Example – Train Tickets Test Cases
Inputs
Test 1
Test 2
Test 3
Test 4
Holds a railcard Y Y N N
Travels after 9 am
Y N Y N
Expected results
Pay £10
Pay £15
Pay £20
Pay £30
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 24
Example – ATM
Conditions
Rule 1
Rule 2
Rule 3
Rule 4
Enter valid card Y Y N N
Enter valid PIN Y N Y N
Actions Auth Reject Reject Reject
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 25
Example – ATM “don’t care”
Conditions
Rule 1
Rule 2
Rule 3
Enter valid card Y Y N
Enter valid PIN Y N ---
Actions Auth Reject Reject
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 26
Example – File Backup
ConditionsRule
1Rule
2Rule
3Rule
4
Stick inserted N Y Y Y
Stick formatted --- N Y Y
Sufficient space --- --- N Y
Actions Fails Fails Fails Works
Pairwise Testing
Acknowledgement
This session on pairwise testing techniques is based on the book by Lee Copeland: A practitioner’s guide to software test
design, published by Artech House, 2004, ISBN 1-58053-791-X
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 32
Pairwise Testing
• Pairwise is based on testing pairs of combinations of variables rather than all combinations
• It reduces the number of test cases• Many faults are either
– Single mode– Double mode
• There are two recognised methods:– Orthogonal arrays– Allpairs algorithm
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 33
Pairwise Example
• Consider a simple database enquiry screen:
– It has 4 fields
– Each field has three possible choices
– Giving 3 X 3 X 3 X 3 = 81 possible combinations to test
• With Pairwise Testing
– You can reduce this to 9 tests!
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 34
Pairwise Example
• Holiday destination– UK, USA, Europe
• Price range (for family of 5)– Budget: Under £2000– Medium: £2000 to £5000– Expensive: Over £5000
• Standard of hotel– Basic, 4 star, luxury
• Availability– Now: Within 7 days from today– Soon: After a week but up to one month from today– Later: Later than one month from today
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 35
Orthogonal Array L9 (34)
A B C D
Row 1 1 1 1 1
Row 2 1 2 2 2
Row 3 1 3 3 3
Row 4 2 1 2 3
Row 5 2 2 3 1
Row 6 2 3 1 2
Row 7 3 1 3 2
Row 8 3 2 1 3
Row 9 3 3 2 1
This orthogonal array has 9 rows, 4 columns and three ‘levels’ per column
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 36
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 1 1 1 1
Row 2 1 2 2 2
Row 3 1 3 3 3
Row 4 2 1 2 3
Row 5 2 2 3 1
Row 6 2 3 1 2
Row 7 3 1 3 2
Row 8 3 2 1 3
Row 9 3 3 2 1
Each variable becomes a column in the orthogonal array
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 37
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 UK 1 1 1
Row 2 UK 2 2 2
Row 3 UK 3 3 3
Row 4 US 1 2 3
Row 5 US 2 3 1
Row 6 US 3 1 2
Row 7 Europe 1 3 2
Row 8 Europe 2 1 3
Row 9 Europe 3 2 1
For each column we fill out the possible choices
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 38
Orthogonal Array L9 (34)
Where Price Hotel When
Row 1 UK Budget Basic Now
Row 2 UK Medium 4 star Soon
Row 3 UK High Luxury Later
Row 4 US Budget 4 star Later
Row 5 US Medium Luxury Now
Row 6 US High Basic Soon
Row 7 Europe Budget Luxury Soon
Row 8 Europe Medium Basic Later
Row 9 Europe High 4 star Now
Now the array is complete
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 39
Orthogonal Array L9 (34)
Where Price Hotel When
Test 1 UK Budget Basic Now
Test 2 UK Medium 4 star Soon
Test 3 UK High Luxury Later
Test 4 US Budget 4 star Later
Test 5 US Medium Luxury Now
Test 6 US High Basic Soon
Test 7 Europe Budget Luxury Soon
Test 8 Europe Medium Basic Later
Test 9 Europe High 4 star Now
Each row becomes a test case – you can add more if you wish
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 40
How to use Orthogonal Arrays
• Identify the variables• Determine the number of choices for each• Locate an orthogonal array of the right size• Map the testing problem onto the array• Construct the test cases
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 41
What size array do we need?
Variable Choices
Browsers IE5, IE6, …. 8 in total
Plug-ins None, Real player, Media Player
Operating systemsW95, W98, W2000, NT, XP, XP-PRO
Server IIS, Apache, WebLogic
Server O/S NT, XP, Linux
How many possible combinations are there?
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 42
Using Orthogonal Arrays
• We require an array with 5 columns– Column 1 needs 8 levels (browsers)– Column 2 would need 3 levels (plug ins)– Column 3 would need 6 levels (O/S)– Columns 4 & 5 would need 3 levels (Server, Server O/S)
• Ideally we need 81 61 33 but it doesn’t exist!
• The next largest is L64 (82 43) – 2 columns of 1 – 8– 3 columns of 1 – 4
• So rather than 1,296 tests we need only 64 tests!• Remember - one test case per row
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 43
Allpairs Algorithm
• Devised by James Bach• Generates fewer tests• In our example, it generates just 48 tests
compared to 64 from using orthogonal arrays• Consider 100 variables, 2 choices
– Using orthogonal arrays you need 101 tests– Using all pairs you need only 10 tests
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 44
Pairwise Testing – Summary
• Test all pairs rather than all combinations– With 4 variables and 3 choices for each you would
require 81 test cases whereas you can cover all pairwise input combinations with only 9 tests
– For 13 variables and 3 choices for each you would require1,594,323 test cases whereas you can cover all pairwise input combinations with only 15 tests
• Pairwise guides us to help find– Single mode defects– Double mode defects
• You should always add any known risky combinations
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 45
Effective Testing – Summary
• Effective Testing uses– A risk-based approach– Formal test case design techniques
• The benefits are– Reduced number of test cases– Repeatable well documented tests– Improved planning, estimation and communication– Facilitates earlier involvement with the business– Find faults earlier– Give you a consistent approach to testing– Save time and money– Enables test automation
SoftTest 16/02/06 Copyright 2006 CaseMaker International Ltd 46
After all, what is quality?
Presented by Stephen Allott
CaseMaker InternationalA Diaz & Hilterscheid group company
Effective Testing
Using industry standard techniques
SoftTest Ireland 16th February 2006