The science behind software testing
© Grid Tools Ltd 2014
HP ALM: Less Test Cases more Coverage
Huw PriceMD Grid Tools Ltd
OR
Huw PriceMD Grid Tools Ltd
How Many Testers does it take to screw in a light bulb?
The Answer is 3!
What is Agile Designer?
Requirements
Clear LogicalSpecs
Storyboards
Expected Results
OutsourceWork
Packages
Test Cases
Data-2
DRAW STORY BOARD TEST CASES TEST DATA CLEAR SPECIFICATIONS
Data andVirtualization
Agile
Outsourcing Data
Flow
Logic
Agenda
• Different Coverage Techniques
• Building the perfect set of test cases from Requirements
• Updating Test Cases when Requirements Change
• Optimizing Existing Test Cases
Common Challenges
• 56% of software defects can be traced back to ambiguities in requirements1
• Detecting and resolving production defects is over 40x more expensive than in requirements2
• Industry standard for manual test case design coverage is between 10-20%
• Over-testing of some functions by factors of 40 is not uncommon
• Change requests to existing software requirements can take up to as much as 60% of the SDLC
(Stats: 1 and 2 - Bender RBT, Others: Grid-Tools)
What is Coverage?
It is not:
• Code Covered
• Number of Test Covered – Tests Run
• Percentage of use cases
• All Paired Combinations
It is:
• Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements
• Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other out - Observability
Copyright 2008 Bender RBT Inc. 9
What is Coverage?
If the customer is a business client or a preferred personal client and they have a checking account, $100,000 or more in deposits, no overdraft protection and fewer than 5 overdrafts in the last 12 months, set up free overdraft protection otherwise do not give overdraft protection.
This function has sixty-four possible combinations of input from which to select test cases:
So which ones to choose?
Different Coverage Techniques
Combinatorial – All Pairs – Constrained All Pairs
• Does not support Expected Results - You have to work it out for each combination which is very time consuming
• Combinatorial is something that ‘accidentally’ increases functional coverage (and only to a point)
• Combinatorial does not give you any solid metrics on how good your testing actually is – you can only infer that x out of y combinations have been covered, but it has no bearing in actual functional logic.
• You end up with lots of false errors that have to be checked by hand
Copyright Bender RBT Inc. 2009 11RBT24261
Cause-Effect Graphing Observable Events and Path Sensitizing
• Assume C and F are not observable events.
• Assume A is stuck at FALSE.• Enter as a test case A(T), B(T),
D(T), E(T).• Results should be C(T), F(T)
and G(T).
T
T
TT
T
T
T
Different Coverage Techniques
Different Coverage Techniques
Optimized Flow Chart Modelling
• Advanced Graph Optimization
• Most projects already have a flow chart
• Similar results to Cause and effect
• Supports Looping
• Supports Constraints
• Not applicable for non sequential based logic
Agile Designer – Test Case Techniques
Coverage Feature Agile Designer
Combinatorial Yes
Flow Chart Yes
Cause and Effect Yes
Constraints Yes
Observability Yes
Looping Yes
Complexity Metrics Yes
Typical Examples of Test Cases built Manually from Requirements
Baseline Agile Designer
Test Cases Created 14 17
Time Taken 5 hours 2 hours
Functional Coverage - % 16% 100%
Baseline Agile Designer
Test Cases Created 150 19
Over Testing x18 x2
Time Taken 5 hours 40 mins
Functional Coverage - % 80% ~100%
Baseline Agile Designer
Test Cases Created 3 12
Time Taken 4 hours 30 mins
Functional Coverage - % 5% 100%
Leg Before Wicket Rule
Demo 1
Visio VersionNode Types
Existing CoverageHP ALM Data Packs
Screw in a Light Bulb
Demo 2
Different OptimizationLooping
Path HintsTesting Importance, risk based
Business Analyst
Programmer
Tester
User
Initial Project
Test Cases
Program
Requirements Business Analyst
Programmer
Tester
User
Change Request
Test Cases
Program
Change request
Business Analyst
Programmer
Tester
User
New Process
Requirements
Tester
Test Cases
Program
RED=HARD WORK
Evolving to Greater Efficiency with Agile Designer
A day at the Races
Demo 3
Changes to RequirementsFix Broken Test Cases
Return to a 100%
The HP Test Case Factory
Baseline Optimized Reduction in Testing
Existing Test Cases 8 9
Test Coverage - % 20% 100%
Functional Nodes 78 31 60%
Edges 76 44
Baseline Optimized Reduction in Testing
Existing Test Cases 15 7
Test Coverage - % 30% 100%
Functional Nodes 113 39 65%
Edges 117 50
N.B. Node = design step in test case / Edge = Journey from one design step to the next
Typical Examples of Optimized Test Cases Built with No Requirements
Leg Before Wicket Rule
Demo 4
Reverse Engineer
Test cases designed manually by hand usually have quite a static flow.
As you can see by these examples there are a few flows which cover the same functionality.
A->B->C->D
A->B->C->E
R->A->B->C->D
Agile Designer Test Cases
Agile Designer Test Cases
Each has its individual starting point and set of steps.
G->F->A->B->C
Q->P->R->G
But often cross paths in terms of Testing the same features several times.
We want to avoid this.
When we have imported these into Agile Designer we can see we have 5 paths but as we mentioned before some cover similar paths.
A occurs 4 timesB occurs 4 timesC occurs 4 timesR occurs 2 timesG occurs 2 times
This gives us the perfect opportunity to optimize and improve the test cases
Agile Designer Test Cases
From this example we have 5 test cases but have we covered each functional step in the most optimal of fashions?
5 Test Cases24 Nodes (Design Steps)22 In/Out Edges (Journeys)
Agile Designer Test Cases
Agile Designer Test Cases
So now we optimized the flow and get something like this.
Optimize:- Removed duplicate design
steps and revaluated how the test cases should link together.
During this evaluation process we are able to navigate through the diagram from Start->End without duplicates causing us problems.
Agile Designer Test Cases
Post Optimize you can see that the number of Nodes/Edges we are testing has reduced significantly.
- 12 Nodes (Design Steps)- 20 Edges (Decisions)
This shows us a 50% reduction in steps we need to take whilst giving us 100% in functional coverage.
We’ve also reduced the number of decisions by 2. This highlights that although a smaller reduction our decision process is much more intelligent and with just 2 fewer we are covering more functions.
Agile Designer makes Continuous Integration = “Idea to Production”
Optimized Tests
Existing Test Cases
Test Data
Automation
Continuous Development
Agile Designer - Summary
Think of a Requirement like a circuit board
• The Test Cases will reflect the requirements
Static Testing
• You can Accurately Measure your Coverage
• You can Automatically create the MinimumTest Cases
With Maximum Coverage
• You can reduce and optimize your existing test cases
• You can predict how long software will take to develop
Agile Designer – Next Steps
• There is a free limited download
• Contact sales for a full copy trial – Linked to HP ALM and Datamaker
• We offer half day training courses (Free T shirt!)http://info.grid-tools.com/agile-designer-training-workshops
• We offer an Agile Designer Certification
• We are looking for partners to roll out the test optimization factory
• We are looking for local partners
Huw PriceMD Grid Tools Ltd