combinatorial black-box testing with classification trees
TRANSCRIPT
W14 Session 10/26/2016 3:00:00 PM
Combinatorial Black-Box Testing with Classification Trees
Presented by:
Peter Kruse
Berner & Mattner Systemtechnik GmbH
Brought to you by:
350 Corporate Way, Suite 400, Orange Park, FL 32073 888---268---8770 ·· 904---278---0524 - [email protected] - http://www.starcanada.techwell.com/
Peter Kruse Berner & Mattner Systemtechnik GmbH Software engineer and test consultant Peter Kruse works in the domain of testing, including evolutionary testing and the classification tree method. He is an experienced software developer and tester in the German automotive industry. Peter's project experience includes hardware-in-the-loop testing, model driven-development, ISO 26262, and evolutionary structural and functional testing. Apart from combinatorial testing, Peter's focuses on improving the costs and benefits in any test process and related test activity, striving to balance emerging trends and ideas with approved methods.
COMBINATORIAL BLACK-BOX TESTING WITH CLASSIFICATION TREES PETER M. KRUSE
Test Process and Test Activities
Testmanagement, Testdokumentation Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
e.g. Module Test, Integration Test, System Test
Specification Tested (Sub-)
System
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?
Test Process and Test Activities
Testmanagement, Testdokumentation Test
Planning
Test
Design
Test
Evaluation
Test
Execution
Test
Implementation
Test Management / Test Documentation
Specification Tested (Sub-)
System
e.g. Module Test, Integration Test, System Test
Solution: Put emphasis here.
Motivation: How to obtain and structure test goals and efforts –
in a reliable, standard-conform, reproducible way?
Classification Tree Method
• Test Design Technique
• Systematic Process
• Graphical Representation of Test Problem
• Independent of Test Domain, Test Level and Test Object
• Input: (Functional) Specification of System under Test
• Output: Set of Tests to be Implemented
• Tool Support: TESTONA
• Recommended for Certified Tester
• Part of ISO 29119-4
Classification Tree Method Example
• Test object: Database Management System
Select a test object: decompose
Classification Tree Method Example
Input domain
Determine input data space
Select a test object: decompose
Classification Tree Method Example
Privileges Operation Access Method
Input domain
Identify relevant aspects (e.g.
from specification)
Determine input data space
Select a test object: decompose
Input domain
Classification Tree Method Example
Privileges Operation Access Method
Classify the input data
space into classes
Classification Tree Method Example
1
2
3
Input domain Privileges Operation Access Method
Combine classes
into test cases
Classify the input data
space into classes
Classification Tree Method – Steps
This is how TESTONA structures test goals:
reliable, standard-conform, and reproducible.
Combine classes into
test cases
Identify relevant
aspects (e.g. from
specification)
Determine input data
space
Select a test object:
decompose
Classify the input data
space into classes
Privileges
Operation
Access Method
Test Generation with TESTONA
• Equivalence Class Testing
• Boundary Value Analysis
• Combinatorial Testing
• Constraints support (dependency rules)
• Minimal Combination (test each input value at least once)
• All-Pairs test (Pairwise)
• Mixed-Strength Generation
• Full Combination (Cartesian product of all input values)
• Prioritized and Statistical Testing
This is how TESTONA obtains test efforts:
reliable, standard-conform, and reproducible.
Combine classes
into test cases
Automate test
generation
TESTONA
TESTONA
• Graphical editor for test design (e.g. classification tree method, boundary value
analysis, equivalence classes, etc.)
• Automatic verification of test cases and dependency rules
• Automatic test case generation
• Tag approach for annotation of any
information and meta-data
• Generic and customized
tool coupling
Used by
EXPECTED TEST OUTCOMES
Test Oracle
William E. Howden (1978): non-trivial challenge of deciding whether a test case has passed or failed
Non-automated Oracles • Trivial case: Manual Assignment
Implicit Oracles • Indirect evaluation (e.g. no exception)
Derived Oracles • A posteriori (e.g. Regression Test)
Specified Oracles • Based on (formal) Specification
Categorization by Barr, Harman, McMinn,
Shahbaz, and Yoo (2015)
Non-automated Oracle Example
• Manual assignment for each test case
Implicit Oracle Example
• Monitor system
No exceptions at
runtime,
CPU load, …
Derived Oracle Example
• Run once, record results, inspect, assign to tests
Specified Oracle Example
(New AND Success ↔ more) AND (Delete AND Success ↔ less)
(empty AND NOT New) OR (full AND New) ↔ Fail
TEST IMPLEMENTATION (TEST SCRIPT GENERATION)
Code Generation: Motivation
Classification
Tree Test Cases
Specification
(Functional Requirements,
Agile Board, etc.)
Code Generation: Motivation
Classification
Tree Test Cases TESTONA
generic
code
export
• Template-based export of test cases to JUnit, XML, text, ...
Specification
(Functional Requirements,
Agile Board, etc.)
Technical Details / Demonstration: JUnit
setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
Include these per mark in test case
@Test
public void %TEST% throws Exception {
}
Include this per test case
Technical Details / Demonstration: JUnit
setDB("%CLASS%");
int oldCount = getCount();
setUser("%CLASS%");
perform("%CLASS%");
assertTrue(getCount() %CLASS% oldCount);
assertEquals(isSuccess(), %CLASS%);
Include these per mark in test case
@Test
public void %TEST% throws Exception {
}
import org.junit.Assert.*;
public class Example {
}
Include this per test case
Include this once at start
Technical Details / Demonstration: JUnit
Technical Details / Demonstration: XML
<database>%CLASS%</database>
<user>%CLASS%</user>
<action>%CLASS%</action>
<expect val="count">%CLASS%</expect>
<expect val="ok">%CLASS%</expect>
Include these per mark in test case
<testcase>
</testcase>
<testspec>
</testspec>
Include this per test case
Include this once at start
Technical Details / Demonstration: XML
CONCLUSION
• Expected result prediction
• Test Script Generation
Conclusion
• Classification Tree Method to structure test goals
• TESTONA to obtain test efforts
Peter M. Kruse @pmk99
www.testona.net
