introduction to softwaretesting
DESCRIPTION
Introduction to SoftwareTesting. CSC 450. Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc. What is software testing?. “Testing is the process of exercising a program with the intent of finding Errors.”. - PowerPoint PPT PresentationTRANSCRIPT
CSC450CSC450
1
• CSC 450
Slides adapted from various sources including Software Engineering: A Practitioner’s Approach, 6/e by R.S. Pressman and Associates Inc.
Introduction to SoftwareTesting
CSC450CSC450
2
CSC450CSC450
3
CSC450CSC450
4
CSC450CSC450
5
CSC450CSC450
6
CSC450CSC450
7
What is software testing?
“Testing is the process of exercising a program with the
intent of finding Errors.”
“Testing is the process of exercising a program with the
intent of proving its correctness.”
CSC450CSC450
8
What is software testing?
“Observing the execution of a software system to validate
whether it behaves as intended and identify potential malfunctions.”
Antonia Bertolino
CSC450CSC450
9
errors
requirements conformance
performance
an indicationof quality
What does testing show?
CSC450CSC450
10
Bugs remaining in the code!
…
What does testing NOT show?
CSC450CSC450
11
CSC450CSC450
12
Why is testing so difficult?
How many paths are there through this loop? How long would it take to execute all paths @
1 test per millisecond (10-3)?
1
32 54 loop < =20 X
ab
c d
e
CSC450CSC450
13
A Testing Scenario
1
32 54 loop < =20 X
ab
c d
e
• There are about 1014 possible paths• How long would it take to execute all paths?
CSC450CSC450
14
A Testing Scenario1
32 54loop < =20 X
ab
c d
e
• How long would it take to execute 1014 possible paths @ 1 paths per 10-3 seconds?– 1 year =31,536,000 seconds ~= 32,000,000,000
milliseconds– Total time = 1014 / 32 * 109 ~= 105 /32 years– About 3,170 years @ one test/millisecond– About 3 years @ one test/microsecond– About 11 days @ one test/nanosecond
CSC450CSC450
15
Why is testing so difficult?
Because exhaustive testing is impossible Exhaustive testing means:
Testing a program with all combinations of input values and preconditions for each element of the software under test and/or
Testing all paths through a program.
132 54
loop <= 20 X
abc d
e
CSC450CSC450
16
Why is testing so difficult?Loop
header
Conditional
statementstatement
Looping : for (i=0; i< n; i++) etc.
• How long would it take to execute all paths @ 1 test per millisecond (10-3)?
CSC450CSC450
17
Terms! Terms! Terms!Anomal
y
Exception
BUG
Error Defect
Failure
Fault
CSC450CSC450
18
Testing Terminology• Fault/defect
– An incorrect step, process, condition, or data definition in a computer program.
– Software problem found after release (Pressman)• Error
– An incorrect internal state resulting from a fault.– A discrepancy between a computed, observed, or measured
value or condition and the true, specified, or theoretically correct value or condition (ISO).
– Software problem found before release (Pressman) • Failure
– The inability of a system or component to perform its required functions within specified performance requirements (IEEE).
• Failures are caused by faults, but not all faults cause failures.
CSC450CSC450
19
Testing Terminology Anomaly
Anything observed in the documentation or operation of software that deviates from expectations based on previously verified software products or reference documents (IEEE).
Bug Things the software does that it is not supposed to do Something the software doesn't do that it is supposed to.
Exception An event that causes suspension of normal program
execution; types include addressing exception, data exception, operation exception, overflow exception, protection exception, underflow exception (IEEE).
CSC450CSC450
20
Testing Terminology: Testing vs. Debugging
Testing Evaluating software to determine conformance
to some objective. Debugging
Finding a fault, given a failure.
CSC450CSC450
21
Testing Terminology: The RIP Fault/Failure Model
Three conditions are necessary for failures to be observable.
Reachability: The location of the fault must be reached.
Infection: After executing the location, the state of the program must be incorrect.
Propagation: The infected state must propagate to cause incorrect program output.
CSC450CSC450
22
Testing Terminology Validation
Does software meet its goals? Are we building the right product? User-centric!
Verification Are we building the product right? E.g., does the
code reflect the design? Developer-centric! Name for the artifact being tested
Implementation under test (IUT) Method under test (MUT), object under test (OUT) Class/component under test (CUT), etc.
CSC450CSC450
23
Testing Terminology: Test Plan A test plan specifies how we will demonstrate
that the software is free of faults and behaves according to the requirements specification
A test plan breaks the testing process into specific tests, addressing specific data items and values
Each test has a test specification that documents the purpose of the test
If a test is to be accomplished by a series of smaller tests, the test specification describes the relationship between the smaller and the larger tests
The test specification must describe the conditions that indicate when the test is complete and a means for evaluating the results
CSC450CSC450
24
Testing Terminology: Test Oracle A test oracle is the set of predicted
results for a set of tests, and is used to determine the success of testing
Test oracles are extremely difficult to create and are ideally created from the requirements specification
CSC450CSC450
25
Testing Terminology: Test Case
A test case is a specification of The pretest state of the IUT
Prefix values: values needed to put software into state required before test execution.
A set of inputs to the system Postfix values: values that need to be sent
to the software after test execution. The expected results
CSC450CSC450
26
Testing Terminology Test Point
A specific value for test case input and state variables.
Domain Set of values that input or state variables of IUT may
take. Heuristics for test point selection
Equivalence classes – one values in set represents all values in set.
Partition testing techniques Boundary value analysis Special values testing
Test suite A collection of test cases.
CSC450CSC450
27
Testing Terminology Fault model
A description about where faults are likely to occur in a program.
Test strategy An algorithm or heuristic for creating test cases from
a representation, an implementation or a test model Test model
A description of relationships between elements of a representation or implementation.
Test design The process of creating a test suite using a test
strategy. Test effectiveness: relative ability of testing
strategy to find bugs. Test efficiency: relative cost of finding bugs.
CSC450CSC450
28
Testing Terminology: Testing Strategies
Responsibility-based test design Behavioural, functional, black-box,
etc. testing Implementation-based test design
Relies on source code,, e.g., white-box Fault-based testing
CSC450CSC450A Classification of Testing: The V Model
Requirements Analysis
Architectural Design
Subsystem Design
Detailed Design
Implementation
Acceptance Test
System Test
Integration Test
Module Test
Unit Test
CSC450CSC450
A Classification of Testing Graphs Logical expressions Input domain characterizations Syntactic descriptions
30
CSC450CSC450
31
Test-Case Exercise
UML class model for Figure Hierarchy
CSC450CSC450
32
The Polygon Classabstract class Polygon {
abstract void draw(int r, int g, int b); /* color closed area*/abstract void erase(); /* set to background rgb */abstract float area(); /* return area */abstract float perimeter(); /* return sum of sides */abstract Point center(); /* return centroid pixel */
}
CSC450CSC450
33
The Triangle Classclass Triangle extends Polygon {
public Triangle(LineSegment a, LineSegment b, LineSegment c){…}public void setA(LineSegment a) {…}public void setB(LineSegment b) {…}public void setC(LineSegment c) {…}
public LineSegment getA() {…}public LineSegment getB() {…}public LineSegment getC() {…}
public boolean is_isoceles(){…}public boolean is_scalene(){…}public boolean is_equilateral(){…}
public void draw(int r, int g, int b){…}public void erase(){…}public float area(){…}public float perimeter(){…}public Point center(){…}
}
CSC450CSC450
34
The LineSegment Classclass LineSegment extends Figure {
public LineSegment(Point x1, Point y1, Point x2, Point y2){…}public void setx1(Point x1) {…}public void sety1(Point y1) {…}public void setx2(Point x2) {…}public void sety2(Point y2) {…}
public Point getx1() {…}public Point gety1() {…}public Point getx2() {…}public Point gety2() {…}
}
CSC450CSC450
35
Test-Case Exercise Specify as many test cases as you can
for the Triangle class defined above. The class is used to determine if a
triangle is: isosceles, equilateral or scalene. A triangle is isosceles if two of its sides are
equal. A triangle is equilateral if all its sides are
equal. A triangle is scalene if its sides are of
different sizes.
CSC450CSC450
36
Test-Case Exercise A valid triangle must meet two
conditions: No side may have a length of zero. Each side must be smaller than the sum
of all sides divided by two. i.e. Given s = (a + b + c)/2, then a < s,
b < s, c < s must hold. Stated differently, a < s && b < s && c <
s a < b + c, b < a + c and c < a + b.
CSC450CSC450
37
Test-Case Exercise Specify as many test cases as you can
for the Triangle class defined above.
Test Case DescriptionTest Case Input
a b c Expected Output
1 Valid isosceles triangle 3 3 4 Isosceles22829303132333435
CSC450CSC450
Solution To Test Case Exercise
CSC450CSC450• Solution To Test Case Exercise
CSC450CSC450• Solution To Test Case Exercise
CSC450CSC450• Solution To Test Case Exercise
CSC450CSC450
42
The C++ IntSet Class Design test cases to test the methods (ignore the constructor and destructor) of
the IntSet class. The class performs operations on a single set – duplications are not allowed If add(val) is called and val is already in the set, then the Duplicate exception is
thrown.
class IntSet {public: //operations on a set of integers
IntSet(); /* constructor */~IntSet(); /* destructor */IntSet& add (int); /* add a member */IntSet& remove (int); /* remove a member */IntSet& clear (int); /* remove all members */int isMember (int arg); /* Is arg a member */int extent (); /* Number of elements */int isEmpty (); /* Empty or not */
} Design test cases to test the methods (ignore the constructor and destructor).
Test Case Design Exercise 2
CSC450CSC450
43
Who Tests the Software?
independent testerdeveloper Understands the
system but, will test "gently“ and, is driven by
"delivery"
Must learn about the system
but, will attempt to break it and, is driven by quality
CSC450CSC450
44
Testing Strategy
We begin by ‘testing-in-the-small’ and move toward ‘testing-in-the-large’
For conventional software The module (component) is our initial focus Integration of modules follows
For OO software “testing in the small” OO class
Has attributes and operations and implies communication and collaboration
CSC450CSC450
Junit Over
view
CSC450CSC450
46
Unit Testing
moduleto betested
results
test cases
softwareengineer
The units comprising a system are individually tested The code is examined for faults in algorithms, data and syntax A set of test cases is formulated and input and the results are evaluated The module being tested should be reviewed in context of the requirements specification
interface local data structuresboundary conditionsindependent pathserror handling paths
CSC450CSC450
47
Unit Test Environment
Module
stub stub
driver
RESULTS
interface
local data structures
boundary conditions
independent paths
error handling paths
test cases
CSC450CSC450
What is JUnit? JUnit is an open source unit testing
framework for Java created by Kent Beck and Erich Gamma.
JUnit features include: Assertions for testing expected results Test fixtures for sharing common test
data Test suites for easily organizing and
running tests Graphical and textual test runners
CSC450CSC450
JUnit Terminology Fixture: a set of objects against which tests are run
A test fixture is ideal when two or more tests are to be executed for a common set of objects. In these scenarios, a test fixture avoids duplicating the initialization tasks before each test and the cleanup activities after each test
setup: a method which sets up the fixture, called before each test is executed.
teardown: a method to tear down the fixture, called after each test is executed.
Test Case: a class which defines the fixture to run multiple tests
- create a subclass of TestCase - add an instance variable for each part of the fixture - override setUp() to initialize the variables and allocate resource - override tearDown() to release any permanent storage, etc.
Test Suite: a collection of test cases.
CSC450CSC450
JUnit 3
CSC450CSC450How To Write JUnit 3 Test Cases Import the Junit framework:
import junit.framework.*; Extend the “TestCase” class
extend junit.framework.TestCase; Make use of special method-name conventions:
Begin each test method name with the prefix ‘test’. Each test case is written in a method that has a name that
begins with “test” Create a mehod with the name setUp to initialize your
object, etc. Create a mehod with the name tearDown to clean-up after
your test. etc.
CSC450CSC450
53
A Case Studyclass Money {
private int fAmount; private String fCurrency;
public Money(int amount, String currency) { fAmount= amount; fCurrency= currency; }public int amount() { return fAmount; } public String currency() { return fCurrency; }
public Money add(Money m) { return new Money(amount()+m.amount(), currency()); }
}
CSC450CSC450
54
How to Write A Simple Standalone TestCase
import junit.framework.*; public class MoneyTest extends TestCase { //… public void testSimpleAdd() {
Money m12 = new Money(12, "Dollar"); // (1)
Money m14 = new Money(14, "Dollar"); Money expected = new Money(26, "Dollar"); Money result = m12.add(m14); // (2) Assert.assertTrue(expected.equals(result)); // (3) } }
1. Creates the objects we will interact with during the test. This testing context is commonly referred to as a test's fixture. All the fixture contains for the testSimpleAdd test are some Money objects.
2. Exercises the objects in the fixture. 3. Verifies the result
CSC450CSC450
55
Assert assertEquals(expected, actual) assertEquals(message, expected, actual) assertEquals(expected, actual, delta) assertEquals(message, expected, actual, delta) assertFalse(condition) assertFalse(message, condition) AssertNotNull(object) AssertNotNull(message, object) AssertNotSame(expected, actual) AssertNotSame(message, expected, actual) assertTrue(condition) assertTrue(message, condition)
CSC450CSC450
56
Start to Use it1. Download the latest version of JUnit from http://download.sourceforge.net/junit/2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath
%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool
to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type:
java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type:
java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type:
java junit.swingui.TestRunner junit.samples.AllTests
Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path
Important: Don't install the junit.jar into the extension directory of your JDK installation.
If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse
CSC450CSC450
57
A Second Examplepublic interface ISorting {
//etc.}public SortingComparison implements ISorting{ //... }
public void bubbleSort(int[] a); //implements bubblesort algorithm
import junit.framework.*; public class SortingTest extends TestCase { SortingComparison obj;//instance variable in
the fixture
public SortingTest(String name) { super(name); }
public void setUp() { obj = new
SortingComparison(); } public void
testSimpleTest() { int answer = 2; assertEquals((1+1),
answer); }
public void testBubble() { //SortingComparison obj = new SortingComparison(); int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0}; int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000}; obj.bubbleSort(qArray); obj.printArray(qArray); obj.printArray(rArray); assertTrue(java.util.Arrays.equals(qArray, rArray));
} public void tearDown () { //do something }
CSC450CSC450
58
Example code 1
import junit.framework.TestCase;public class JunitExamples extends TestCase { SortingComparison obj;//instance variable in the fixture public JunitExamples(String name) { super(name); } public void setUp() { obj = new SortingComparison(); } public void tearDown () { //do something } public void testBubble(){ //SortingComparison obj = new SortingComparison(); int[] qArray = {1,9,8,7,600000,5,4,3,2,10,0}; int[] rArray = {0,1,2,3,4,5,7,8,9,10,600000}; obj.bubbleSort(qArray); obj.printArray(qArray); obj.printArray(rArray); assertTrue(java.util.Arrays.equals(qArray, rArray)); }//end of method}//end of class
CSC450CSC450
59
Example code 2
import junit.framework.TestCase;public class JunitExamples extends TestCase { SortingComparison obj;//instance variable in the fixture public JunitExamples(String name) { super(name); } public void setUp() { obj = new SortingComparison(); } public void tearDown () { //do something } public void testIsosceles (){ int side1 =3, side2 = 3, side3 = 0; boolean result = is_isosceles(side1, side2, side3); assertTrue(result == true); }//end of method}//end of class
CSC450CSC450
Running JUnit java junit.textui.TestRunner JavaClassfileName java junit.swingui.TestRunner
JavaClassfileName java junit.awtui.TestRunner JavaClassfileName Use eclipse
Download the JUnit and integrate in eclipse http://
people.uncw.edu/simmondsd/courses/Current/csc450/lectures/14-testing/JUnit%20Exercise/usingJUnitinEclipse.htm
http://www.junit.org/ Create your java project and the java application to
be tested. then … (next slide)
CSC450CSC450
JUnit in Eclipse To create a test
class, select File New Other... Java, JUnit, TestCase and enter the name of the class you will test
Fill this in
This will be filled in automatically
CSC450CSC450
Running JUnitFirst, select a Test classSecond, use this
pulldown menu
Third, Run As JUnit Test
CSC450CSC450
Results Your results are here
CSC450CSC450
64
Start to Use it1. Download the latest version of JUnit from http://download.sourceforge.net/junit/2. Installation unzip the junit.zip file add junit.jar to the CLASSPATH. For example: set classpath=%classpath
%;INSTALL_DIR\junit3\junit.jar 3. Testing Test the installation by using either the batch or the graphical TestRunner tool
to run the tests that come with this release. All the tests should pass OK. for the batch TestRunner type:
java junit.textui.TestRunner junit.samples.AllTests for the graphical TestRunner type:
java junit.awtui.TestRunner junit.samples.AllTests for the Swing based graphical TestRunner type:
java junit.swingui.TestRunner junit.samples.AllTests
Notice: The tests are not contained in the junit.jar but in the installation directory directly. Therefore make sure that the installation directory is on the class path
Important: Don't install the junit.jar into the extension directory of your JDK installation.
If you do so the test class on the files system will not be found. JUnit plug-in for Eclipse
CSC450CSC450
JUnit 4
CSC450CSC450Comparing JUnit 3 to JUnit 4 All the old assertXXX methods are the
same Most things are about equally easy
JUnit 4 makes it easier to test that exceptions are thrown when they should be
JUnit 4 can still run JUnit 3 tests JUnit 4 provides protection against
infinite loops JUnit 4 has some additional features
CSC450CSC450
Migrating to JUnit 4 JUnit 4 requires Java 5 or newer Don’t extend junit.framework.TestCase; just
use an ordinary class Import org.junit.* and org.junit.Assert.*
Use a static import for org.junit.Assert.* Static imports replace inheritance from
junit.framework.TestCase Use annotations instead of special method
names: Instead of a setUp method, put @Before before
some method Instead of a tearDown method, put @After before
some method Instead of beginning test method names with ‘test’,
put @Test before each test method
CSC450CSC450
68
Testing
Strategies
CSC450CSC450
69
Testing Strategyunit test integration
test
validationtest
systemtest
Black Box Testing White Box Testing
CSC450CSC450
70
Unit Testing
moduleto betested
results
test cases
softwareengineer
The units comprising a system are individually tested
The code is examined for faults in algorithms, data and syntax
A set of test cases is formulated and input and the results are evaluated
The module being tested should be reviewed in context of the requirements specification
interface local data structuresboundary conditionsindependent pathserror handling paths
CSC450CSC450
71
Unit Test Environment
Module
stub stub
driver
RESULTS
interface
local data structures
boundary conditions
independent paths
error handling paths
test cases
CSC450CSC450
72
Integration Testing
The goal is to ensure that groups of components work together as specified in the requirements document
Four kinds of integration tests exist Structure tests Functional tests Stress tests Performance tests
CSC450CSC450
73
Integration Testing Strategies
Options:• the “big bang” approach• an incremental construction strategy
CSC450CSC450
74
Top Down Integrationtop module is tested with stubs
stubs are replaced one at a time, "depth first"
as new modules are integrated, some subset of tests is re-run
A
B
C
D E
F G
CSC450CSC450
75
Bottom-Up Integration
drivers are replaced one at a time, "depth first"
worker modules are grouped into builds and integrated
A
B
C
D E
F G
cluster
CSC450CSC450
76
Sandwich TestingTop modules aretested with stubs
Worker modules are grouped into builds and integrated
A
B
C
D E
F G
cluster
CSC450CSC450
77
System Testing The goal is to ensure that all
elements that interact to solve the problem do so correctly. Software, databases, hardware,
people, procedures. Overall functionality and performance
must be achieved.
CSC450CSC450
78
Validation Testing The goal is to ensure that the
system actually does what the customer expects it to do – fulfils requirements
Testing is carried out by customers mimicking real world activities
Customers should also intentionally enter erroneous values to determine the system behavior in those instances
CSC450CSC450
79
Black Box Testing Testing the functionality of the program
The tester either knows nothing about the internal structure of the code or otherwise ignores this information
Test cases are formulated based on expected output of methods
Tester generates test cases to represent all possible situations in order to ensure that the observed and expected behaviour is the same
CSC450CSC450
80
White Box Testing
The tester uses knowledge of the programming constructs to determine the test cases to use
If one or more loops exist in a method, the tester would wish to test the execution of this loop for 0, 1, max, and max + 1, where max represents a possible maximum number of iterations
Similarly, conditions would be tested for true and false
CSC450CSC450
81
High Order Testing
Validation testing Focus is on software requirements
System testing Focus is on system integration
Alpha/Beta testing Focus is on customer usage
Recovery testing forces the software to fail in a variety of ways and
verifies that recovery is properly performed Security testing
verifies that protection mechanisms built into a system will, in fact, protect it from improper penetration
Stress testing executes a system in a manner that demands
resources in abnormal quantity, frequency, or volume
Performance Testing test the run-time performance of software within the
context of an integrated system
CSC450CSC450
82
Strategic Issues
State testing objectives explicitly. Understand the users of the software and develop a
profile for each user category. Develop a testing plan Build “robust” software that is designed to test itself Use effective formal technical reviews as a filter prior to
testing Conduct formal technical reviews to assess the test
strategy and test cases themselves. Develop a continuous improvement approach for the
testing process.
CSC450CSC450
83
Testing Steps
Determine what the test is supposed to measure
Decide how to carry out the tests Develop the test cases Determine the expected results of
each test (test oracle) Execute the tests Compare results to the test oracle
CSC450CSC450
84
Object-Oriented Testing
begins by evaluating the correctness and consistency of the OOA and OOD models
testing strategy changes the concept of the ‘unit’ broadens due to
encapsulation integration focuses on classes and their execution
across a ‘thread’ or in the context of a usage scenario
validation uses conventional black box methods test case design draws on conventional
methods, but also encompasses special features
CSC450CSC450
85
Broadening the View of “Testing”
“It can be argued that the review of OO analysis and design models is especially useful because the same semantic constructs (e.g., classes, attributes, operations, messages) appear at the analysis, design, and code level. Therefore, a problem in the definition of class attributes that is uncovered during analysis will circumvent side effects that might occur if the problem were not discovered until design or code (or even the next iteration of analysis)” - Pressman.
CSC450CSC450
86
OOT Strategy
class testing is the equivalent of unit testing operations within the class are tested the state behavior of the class is examined
integration applied three different strategies thread-based testing—integrates the set of
classes required to respond to one input or event
use-based testing—integrates the set of classes required to respond to one use case
cluster testing—integrates the set of classes required to demonstrate one collaboration
CSC450CSC450
87
Smoke Testing
A common approach for creating “daily builds” for product software
Smoke testing steps: Software components that have been translated
into code are integrated into a “build.” A build includes all data files, libraries, reusable
modules, and engineered components that are required to implement one or more product functions.
A series of tests is designed to expose errors that will keep the build from properly performing its function.
The intent should be to uncover “show stopper” errors that have the highest likelihood of throwing the software project behind schedule.
The build is integrated with other builds and the entire product (in its current form) is smoke tested daily.
The integration approach may be top down or bottom up.
CSC450CSC450
88
Debugging: A Diagnostic Process
CSC450CSC450
89
The Debugging Processtest cases
results
Debugging
suspectedcauses
identifiedcauses
corrections
regressiontests
new testcases
CSC450CSC450
90
Debugging Efforttime requiredto diagnose thesymptom anddetermine thecause
time requiredto correct the errorand conductregression tests
CSC450CSC450
91
Symptoms & Causes
symptomcause
symptom may disappear when another problem is fixed
cause may be due to a combination of non-errors
cause may be due to a system or compiler error
cause may be due to assumptions that everyone believes
symptom and cause may be geographically separated
symptom may be intermittent
CSC450CSC450
92
Consequences of Bugs
damage
mild annoying
disturbingserious
extremecatastrophic
infectious
Bug Type
Bug Categories: function-related bugs, system-related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards violations, etc.
CSC450CSC450
93
Debugging Techniques
backtracking
brute force / testing
Cause-elimination Induction (hypothesis cause)
deduction (cause hypothesis)
CSC450CSC450
94
Debugging: Final Thoughts
symptom you're seeing.Don't run off half-baked, think about the 1.
more insight.Use tools (e.g., dynamic debugger) to gain 2.
If at an impasse, get help from someone else.3.
Be absolutely sure to conduct regression tests when you do "fix" the bug.
4.
CSC450CSC450
95
White-Box
Testing &
Control Flow
Graphs
CSC450CSC450
96
Testing Strategyunit test integration
test
validationtest
systemtest
Black Box TestingWhite Box Testing
CSC450CSC450
97
White-box testing: control-flow graph Graph?
Directed graph? A control flow graph of a
program is a graph that shows the paths that may be traversed when the program is executed. Vertices are statements in the
program
CSC450CSC450
98
Control-flow Graph int n = 0; do{ if(n++ % 2 == 1)
dBase[i]++; else
if(dBase[i] > 100) dBase[i] -= n;else dBase[i] += n;
}while(n < 20);
1
2
34
5 6
7
8
1
2
4
3
5
6
7
CSC450CSC450
99
Cyclomatic Complexity – V(G)
If G is the control flow graph of program P and G has e edges and n nodes, then the cyclomatic complexity of P, V(G) is given by: V(G) = e – n + 2 Or V(G) = the number of decision
nodes + 1
CSC450CSC450
100
Cyclomatic Complexity – V(G)
cyclomatic complexity V(G) = e – n + 2
= 11 – 7 = 4 Or V(G) = the number
of decision nodes + 1 = 3 + 1 = 4
1
2
34
5 6
7
8
CSC450CSC450
101
Cyclomatic ComplexityA number of industry studies have indicated
that the higher V(G), the higher the probability or errors.
V(G)
modules
modules in this range are more error prone
CSC450CSC450
102
Cyclomatic Complexity – V(G)
The cyclomatic complexity is a measure of the number of linearly independent paths in the graph.
An independent path must move along at least one edge not previously traversed
What are the independent paths for this graph?
1-2-3-6-7-8 1-2-3-5-7-8 1-2-4-7-8 1-2-4-7-2…8
1
2
34
5 6
7
8
CSC450CSC450
103
White-box testing: Code coverage White box testing methods emphasize code
coverage Statement coverage
Every statement is executed at least once Branch coverage
Every branch is exercised at least once i.e. every path from a node is traversed at least once
Once true and once false evaluation Multiple condition coverage
All combinations of simple conditions are exercised Basis-path model
Exercise V(G) independent paths Data flow coverage
Evaluation of state of instance variables.
CSC450CSC450
104
Statement Coverage1
2
34
5 6
7
8
Write enough test cases to execute each statement al least once, 1-2-3-5-7-8 1-2-3-6-7-8 1-2-4-7-8
May miss bugs for complex predicates and loops
CSC450CSC450
105
Statement Coverage1
2
34
5 6
7
8
int num = 0;do {
if(num > 5 && (sum-num)<findMax(num)) {if(transform(num)) transpose(Math.pow(num, 2))else sum += num + 5;
}else
sum += findMax(num);
}while(num != getValue(num));
12
4
3
5
6
7
CSC450CSC450
106
Branch Coverage1
2
34
5 6
7
8
Determine number of branches Compute path for each branch
1-2-3-5-7-2 1-2-3-6-7-8 1-2-4-7-8
Write test cases to exercise each path
CSC450CSC450
107
Multiple Condition Coverage
1
2
34
5 6
7
8
Determine true/false combinations for conditions
Write test cases to exercise each combination
The required test cases may be determined using the truth table for the logical operator of the condition.
CSC450CSC450
108
Multiple Condition Coverage
if(num > 5 || isValid(num) && (sum-num<findMax(num) || sum = 100) && isDivisible(num, 7)
Test Case # num > 5 isValid(num) (sum-num) < findMax(num)
sum=100 isDiv (num, 7)
1 F F F2 F T F F F3 F T F T F F4 F T F T T T5 F T T F F6 F T T T T7 T F F F8 T F T F F9 T F T T T10 T T F F11 T T T T
CSC450CSC450
109
Basis-path Testing1
2
34
5 6
7
8
Compute the cyclomatic complexity
Derive independent paths 1-2-3-5-7-8 1-2-3-6-7-8 1-2-4-7-8 1-2-4-7-2…7-8
Write test cases to exercise each path
CSC450CSC450
110
Example 1 V(G) = 2 + 1 = 3 Statement coverage requires 2
test cases Branch coverage requires 2 test
cases Basis-path coverage requires 3
test cases There are four distinct entry-exit
paths 1-2-4-6-7 1-2-4-5-7 1-3-4-5-7 1-3-4-6-7 Which should be chosen?
1
2
4
3
5 6
7
CSC450CSC450
111
Example 2
What is V(G)? How many test cases are
required for statement coverage? How many test cases are
required for branch coverage? How many test cases are
required for basis-path coverage? How many unique entry-exit
paths exist? LIST THE PATHS
A
D
E F
K L
CSC450CSC450
112
Introduction to Mutation Testing & Program Perturbation
CSC450CSC450
113
What is Mutation Testing? Mutation Testing is a testing
technique that focuses on measuring the adequacy of test cases.
Mutation Testing is NOT a testing strategy like path or data-flow testing. It does not outline test data selection criteria.
Mutation Testing should be used in conjunction with traditional testing techniques, not instead of them.
CSC450CSC450
114
Mutation Testing Faults are introduced into a program by
creating many versions of the program Each version is called a mutant. Each mutant contains a single fault.
Test cases are applied to the original program and to the mutant program.
The goal is to cause the mutant program to fail, thus demonstrating the effectiveness of the test case.
CSC450CSC450
115
Example of a Program Mutation
1 int max(int x, int y)2 {3 int mx = x;4 if (x > y)5 mx = x;6 else7 mx = y;8 return mx;9 }
1 int max(int x, int y)2 {3 int mx = x;4 if (x < y)5 mx = x;6 else7 mx = y;8 return mx;9 }
CSC450CSC450
116
Mutation Testing Process Use mutation operators to introduce errors into
the source program One error per mutant
Validate the test suite with unmodified program (insures the test suite functions properly)
Run test suite on mutant programs Analyze result to determine how many non-
equivalent mutants were killed Fix test suite to kill all live mutants until
Mutation Score = 1.0 or specified a threshold score is reached e.g. 0.96
CSC450CSC450
117
Mutation Testing Process
Figure 2. Mutation Testing Flowchart, Source: Mutation 2000, pg 3
Process (cont.)
CSC450CSC450
118
Test Case Adequacy A test case is adequate if it is useful in
detecting faults in a program. A test case can be shown to be
adequate by finding at least one mutant program that generates a different output than does the original program for that test case.
If the original program and all mutant programs generate the same output, the test case is inadequate.
CSC450CSC450
119
Terminology Mutation testing
The creation of a set of mutant programs of the program being tested.
Mutation a single syntactic change that is
made to a program statement. Each mutant differs from the original
program by one mutation.
CSC450CSC450
120
Terminology Mutant
A faulty program; the original with one fault introduced
Mutant Killed Test suite detects the mutant
Mutation Score #of killed mutants/ # of non-equivalent
mutants Mutation Operator
The function that changes the source code to create a mutant
CSC450CSC450
121
Mutation Operators Replace each operand with every
other syntactically correct operand Modify expressions by replacing
operators and inserting new operators
Delete entire statements
CSC450CSC450
122
Categories of Mutation Operators Operand Replacement
Operators: Replace a single operand with
another operand or constant. E.g., if(x > y) if (5 > y) Replacing x by constant 5. if (x > 5) Replacing y by constant 5. if (y > x) Replacing x and y with each
other. E.g., if all operators are {+,-,*,**,/}
then the following expression a = b * (c - d) will generate 8 mutants:
4 by replacing * 4 by replacing -.
CSC450CSC450
123
Categories of Mutation Operators
Expression Modification Operators: Replace an operator or insert new
operators. E.g., if (x == y) if (x >= y) Replacing == by >=. if (x == ++y) Inserting ++.
CSC450CSC450
124
Statement Modification Operators: E.g.,
Delete the else part of the if-else statement.
Delete the entire if-else statement.
Replace line 3 by a return statement.
Categories of Mutation Operators
1 int max(int x, int y)2 {3 int mx = x;4 if (x > y)5 mx = x;6 else7 mx = y;8 return mx;9 }
return x;
CSC450CSC450
125
Mutation System Example The Mothra mutation system for
FORTRAN77 supports 22 mutation operators. E.g., absolute value insertion constant for array reference replacement GOTO label replacement RETURN statement replacement statement deletion unary operator insertion logical connector replacement comparable array name replacement
CSC450CSC450
126
Why Does Mutation Testing Work?
The operators are limited to simple single syntactic changes on the basis of the competent programmer hypothesis.
CSC450CSC450
127
The Competent Programmer Hypothesis Programmers are generally very
competent and do not create “random” programs.
For a given problem, a programmer, if mistaken, will create a program that is very close to a correct program.
An incorrect program can be created from a correct program by making some minor change to the correct program.
CSC450CSC450
128
Mutation Testing Algorithm Generate program test cases. Run each test case against the
original program. If the output is incorrect, the program
must be modified and re-tested. If the output is correct go to the next
step ... Construct mutants
E.g. using a tool like Mothra.
CSC450CSC450
129
Mutation Testing Algorithm (Cont’d)
Execute each test case against each alive mutant. If the output of the mutant differs from the
output of the original program, the mutant is considered incorrect and is killed.
Two kinds of mutants survive: Functionally equivalent to the original
program: Cannot be killed. Killable: Test cases are insufficient to kill
the mutant. New test cases must be created.
CSC450CSC450
130
Mutation Score The mutation score for a set of test
cases is the percentage of non-equivalent mutants killed by the test data.
Mutation Score = 100 * D / (N - E) D = Dead mutants N = Number of mutants E = Number of equivalent mutants
A set of test cases is mutation adequate if its mutation score is 100%.
CSC450CSC450
131
Evaluation Theoretical and experimental results
have shown that mutation testing is an effective approach to measuring the adequacy of test cases.
The major drawback of mutation testing is the cost of generating the mutants and executing each test case against them. Very expensive to run (time, processor
intensive, memory)
CSC450CSC450
132
Program Perturbation Program Perturbation is a
technique to test a program’s robustness.
It is based on unexpectedly changing the values of program data during run-time.
CSC450CSC450
133
Software Failure Hypothesis
Program perturbation is based on the three part software failure hypothesis: Execution: The fault must be
executed. Infection: The fault must change the
data state of the computation directly after the fault location.
Propagation: The erroneous data state must propagate to an output variable.
CSC450CSC450
134
Program Perturbation Process The tester must:
inject faults in the data state of an executing program;
trace the impact of the injected fault on the program’s output.
The injection is performed by applying a perturbation function that changes the program’s data state.
CSC450CSC450
135
The Perturbation Function The perturbation function is a
mathematical function that: takes a data state as its input; changes the data state according to
some specified criteria; produces a modified data state as
output.
CSC450CSC450
136
The Fault Injection A program location N is chosen along
with a set of input variables I that are in scope at location N.
The program is executed until location N.
When execution arrives at location N, the resulting data state is changed (perturbed).
The subsequent execution will either fail or succeed.
CSC450CSC450
137
Program Perturbation Example
Assume the following perturbation function:
1. int perturbation (int x)2. {3. int newX;4. newX = x + 20;5. return newX;6. }
CSC450CSC450
138
Example of a Fault Injection
1. main()2. {3. int x;4. x = ReadInt();5. if (x > 0)6. printf(“X positive”);7. else8. printf(“X negative”);9. }
1. main()2. {3. int x;4. x = ReadInt();4.1 x = perturbation(x);5. if (x > 0)6. printf(“X positive”);7. else8. printf(“X negative”);9. }
CSC450CSC450
139
What Perturbation Testing is and is Not
Perturbation testing is NOT a testing technique that outlines test selection and coverage criteria.
Rather, perturbation testing is a technique that can be used to measure the reliability of the software (tolerance to faults).
CSC450CSC450
140
Evaluation The program is repeatedly
executed and injected with faults during each execution.
The ratio of the number of failures detected divided by the total number of executions is used to predict failure tolerance.
CSC450CSC450
Summary & The End
141
______________________Devon M. Simmonds
Computer Science DepartmentUniversity of North Carolina Wilmington
____________________________________________ _________________
Qu es ti ons?
CSC450CSC450
142
Why is testing so difficult?
There are about 1014 possible paths How long would it take to execute all paths?
About 3,170 years @ one test/millisecond
1
32 54 loop < =20 X
ab
c d
e
CSC450CSC450
143
Example 2 What is V(G)? - 7 How many test cases are
required for statement coverage? - 4
How many test cases are required for branch coverage? - 4
How many test cases are required for basis-path coverage? - 7
How many unique entry-exit paths exist? – 16
A
D
E F
K L – ABDEGKMQS
– ABDEGKNQS
– ABDEHKMQS
– ABDEHKNQS
– ACDEGKMQS
– ACDEGKNQS
– ACDEHKMQS
– ACDEHKNQS
– ACDFJLPRS– ACDFJLQRS– ACDFILPRS– ACDFILQRS– ABDFJLPRS– ABDFJLQRS– ABDFILPRS– ABDFILQRS
CSC450CSC450
144
Example 2 ABDEGKMQ
S ABDEGKNQ
S ABDEHKMQ
S ABDEHKNQ
S ACDEGKMQ
S ACDEGKNQ
S ACDEHKMQ
S ACDEHKNQ
S
A
D
E F
K L– ACDFJLPRS– ACDFJLQRS– ACDFILPRS– ACDFILQRS– ABDFJLPRS– ABDFJLQRS– ABDFILPRS– ABDFILQRS
CSC450CSC450
145
Structure setUp() Storing the fixture's objects in instance
variables of your TestCase subclass and initialize them by overriding the setUp method
tearDown() Releasing the fixture’s
run() Defining how to run an individual test case. Defining how to run a test suite.
testCase()