testing basics dr. eng. amr t. abdel-hamid netw 703 winter 2010 protocol design
TRANSCRIPT
Testing Basics
Dr. Eng. Amr T. Abdel-Hamid
NETW 703
Winter 2010
Pro
toco
l Desig
n
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Program Testing or, Program Simulation
What is software testing?Running a programIn order to find faults
a.k.a. defectsa.k.a. errors a.k.a. flawsa.k.a. faultsa.k.a. BUGS
This enables us to Increase our confidence that the program has high
quality and low risk Because we can never be sure we caught all bugs
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Testing
What isn’t software testing?Purely static analysis: examining a program’s
source code or binary in order to find bugs, but not executing the program
Verification & Validation: Based on building a model and checking it.
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Why Testing?
Ideally: we prove code correct, using formal mathematical techniques (with a computer, not chalk)
Extremely difficult: for some trivial programs (100 lines) and many small (5K lines) programs
Simply not practical to prove correctness in most cases – often not even for safety or mission critical code
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Unit, Integration, System Testing Stages of testing
Unit testing is the first phase, done by developers of modules
Integration testing combines unit tested modules and tests how they interact
System testing tests a whole program to make sure it meets requirements
“Design testing” is testing prototypes or very abstract models before implementation – seldom mentioned, but when possible it can save time.
Model checking may be possible at this stage Functional Testing tests a program from a “user’s” perspective –
does it do what it should?Opposed to unit testing, which often proceeds from the perspective of other parts of the program
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Faults, Errors, and Failures
Fault: a static flaw in a programWhat we usually think of as “a bug”
Error: a bad program state that results from a faultNot every fault always produces an error
Failure: an observable incorrect behavior of a program as a result of an errorNot every error ever becomes visible
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
To Expose a Fault with a Test
Reachability: the test much actually reach and execute the location of the fault
Infection: the fault must actually corrupt the program state (produce an error)
Propagation: the error must persist and cause an incorrect output – a failure
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Example 1
int findLast (int a[], int n, int x) {// Returns index of last element in a// equal to x, or -1 if no such.// n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1;}
Find the fault
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Example 1 (cont.)
int findLast (int a[], int n, int x) {// Returns index of last element in a// equal to x, or -1 if no such.// n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1;}
Here’s a test case:a = {}n = 0x = 2
Everything is ok, as we did not reach thebug
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Example 1 (cont.)
int findLast (int a[], int n, int x) {// Returns index of last element in a// equal to x, or -1 if no such.// n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1;}
Here’s another:a = {3, 9, 4}n = 3x = 2
Reaches the faultInfects state with errorBut no failure
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Example 1 (cont.)int findLast (int a[], int n, int x) {// Returns index of last element in a// equal to x, or -1 if no such.// n is length of a int i; for (i = n-1; i > 0; i--) { if (a[i] = x) return i; } return -1;}
And finally:a = {2, 9, 4}n = 3x = 2
Reaches the faultInfects state with errorAnd fails – returns -1instead of 0
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Controllability and Observability
Goals for a test case:Reach a faultProduce an errorMake the error visible as a failure
In order to make this easy the program must be controllable and observableControllability:
How easy it is to drive the program where we want to go
Observability:How easy it is to tell what the program is doing
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Black Box Testing
Black box testingTreats a program or system as a black box i.e. testing that does not look at source code or internal
structure of the systemSend a program a stream of inputs, observe the
outputs, decide if the system passed or failed the test
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Conformance Testing
How do we test finite state machines?Let’s say we have
Known FSM AKnow all states and transitions
Unknown FSM B (same alphabet)Can only perform experiments
How do we tell if A = B?
Known as the conformance testing or equivalence testing problem
a
c
a
ba
d
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Conformance Testing
Exhaustive tests can be very expensive In general, we cannot computationally afford to perform
complete testingWe will always face the risk of missing errorsEven when we reduce our problem to the simplest
modelThe complexity of testing full equivalence to a
reference model is simply too highExhaustion is exhausting
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Protocol Conformance Testing To confirm if an implementation conforms to its standard Issue1: preparation of conformance tests in coverage of all design
aspects Issue2: time required to run test should not be unacceptably long Two main limitations
Controllability: the IUT cannot be directly put into a desired state, usually requiring several additional state transitions
Observability: prevents the external tester from directly observing the state of the IUT, which is critical for a test to detect errors
Formal conformance testing techniques based on FSM Generate a set of input sequences that will force the FSM
implementation to undergo all specified transitions Black box approach: only the outputs generated by the IUT (upon
receipt of inputs) are observable to the external tester
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Fault Model for FSM Output fault: the machine provides an output different from the
one specified by the output function Transfer fault: the machine enters a different state than that
specified by the transfer function Transfer faults with additional states: number of states of the
system is increased by the presence of faults, additional states is used to model certain types of errors
Additional or missing transitions: one basic assumption is that the FSM is deterministic and completely defined (fully specified). So the faults occur when it turns out to be non-deterministic and/or incompletely (partially) specified
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
FIFO input queues Fault Models Ordering fault: FIFO ordering is not preserved, or in case of
multiple input queues, some input event enters a wrong input queue.
Maximum length fault: the maximum length implemented is less than the one specified, or if an input event gets lost while queue is not overflow.
Flow control fault: errors of ordering or of loss occur, in case the number of submitted input events overflows the maximum queue length specified
Some other definitions & assumptions Deterministic FSM: predictable behavior in a given state for a
given input Strongly connected: for each state pair (si, sj) there is a transition
path going from si to sj , I.e. each state can be reached from any other state
Fully specified: form each state it has a transition for each input symbol. Otherwise partially specified
Minimal: the number of states of M is less than or equal to the number of states of any equivalent machine
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
19/25
Transition Level Approach The methods for protocol conformance test sequence generation Produce a test sequence which checks the correctness of each
transition of the FSM implementation By no means exhaustive, I.e. no guarantee to exhibit correct
behavior given every possible input sequence. The intent is to design a test sequence which guarantees “beyond a
reasonable doubt” Three basic steps for checking a transition:
Step 1: The FSM implementation is put into state si (e.g. reset+transfer)Difficulty in realizing this is due to the limited controllability of
the implementation Step 2: Input ak is applied and the output is checked to verify
that it is ol, as expected; Step 3: The new state of the FSM implementation is checked to
verify that it is sj, as expectedDifficulty in verifying this is due to the limited observability of
the implementation
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
20/25
T-Method: Transition Tour Method
For a given FSM S, a transition tour is a sequence which takes the FSM S from the initial state s0, traverses every transition at least once, and returns to the initial state s0.
Straightforward and simple scheme New state of the FSM is not checked Detects all output errors There is no guarantee that all transfer errors can be detected The problem of generating a minimum-cost test sequence using
the transition tour method is equivalent to the so-called “Chinese Postman” problem in graph theory
First studied by Chinese mathematician Kuan Mei-Ko ( 管梅谷 ) in 1962
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
T-Method EX1
Specification
Transition Tour: b b b a a a
y y x x y x
Imp. 1
Imp. 2
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 2
Transition Tour Sequence:
r, a,
r, c,
r, c, a, b,
r, c, a, b, b,
r, c, b, a, a
r, c, a, c, b, a, c, a,
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Outline
Generate a Transition Tour Sequence for the specification below:
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
D-Method: Distinguishing Sequences A sequence of inputs is a distinguishing sequence (DS) for an
FSM S, if the output sequence produced by the FSM S in response to the input sequence is distinct for each initial state
A DS is used as a state identification sequence A DS is a useful tool for achieving Step 3 in checking the new
state Fault detection power
Detects all output errors Detects all transfer errors
Two severe drawbacks In practice, very few FSMs actually possess a DS Even if an FSM does have a DS, the upper bound on the
length of the DS will be too large to be useful in general The requirement is too strong (leading to W- & U- methods…)
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
D-Method Ex. 1
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 1 The test cases
( -sequences) are:
state 1: r,a,b,b r,b,b,b
state 2: r,b,a,b,b r,b,b,b,b
Test case structure corresponding to 3-steps:preamble, tested transition, state identificationTransfer sequence (Preamble): the minimum cost (shortest path) input sequence taking FSM from one state to another.
state 3: r,a,a,b,b r,a,b,b,b
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 3
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 4 Find DS and -sequences
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
W-Method: Characterizing Sequences
For FSMs that do not possess a DS, W-Method defines partial DS each of which distinguishes a state si from a subset of the remaining states instead of from every state of the FSM
The states of the FSM are first partitioned into blocks which can be distinguished by observing the sequence of outputs produced by a sequence of inputs
Each block is subsequently partitioned into distinguishable sub-blocks, and so on, until each block consists of exactly one state
Main idea is to iteratively find a DS for each subset To identify a state (for step 3)
Applying an input sequence Returning to the state via a transfer sequence Applying a second input sequence, and so on
The complete set of such input sequences for an FSM is called the characterizing set Attach each CS in the set to the end of each transfer
sequence
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 1
For the input sequence Acs1 = A,A, the response is identical for states 2 and 3 (01), but is distinct from that for states 0(00), 1(11), 4(10).
Another input sequence Acs2 = B is distinct for states 2(0) and 3(1).
Acs1 is required to identify states 0, 1, 4, and two input sequences
Acs1 and Acs2, along with appropriate transfer sequences, are required to identify states 2 and 3.
W = {AA, B}
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex2 Find W
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
U-Method: Unique Input/Output Sequences
An I/O behavior that is not exhibited by any other state of the FSM Answer the question of “is the implementation currently in state x?” Advantages against DS & CS
Cost is never more than DS and in practice is usually much less (shorter)
Nearly all FSMs have UIO sequences for each state DS – same for all states; UIO sequence – normally different for
each state To check state s by using UIO sequence of s
Apply input part of UIO, compare output sequence with the expected one
If the same, then the FSM is in the state s; otherwise, not in the state s
If not in state s, no information about the identity of the actual state s’
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Ex. 1
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Analysis
Fault Testing Coverage Fault coverage for D-, W-, and U-methods is better than of T-
method Fault coverage for D-, W-, and U-methods are the same
All of these four methods assume minimal, strongly connected and fully specified Mealy FSM model of protocol entities
On average, T-method produces the shortest test sequence, W-method the longest. D- and U- methods generate test sequence of comparable lengths
T-method test sequences are able to detect output faults but not transition
D-, W-, and U-methods are capable of detecting all kinds of faults and give the same performance.
U-method attracts more and more attentions and there are several approaches based on the basic idea with some improvements
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
White Box Testing
Opens up the box! (also known as glass box, clear box, or structural
testing)
Use source code (or other structure beyond the input/output spec.) to design test cases
Brings us to the idea of coverage
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Coverage Measures
In general, used to measure the quality of a test suiteEven in cases where the suite was designed for some
other purpose (such as testing lots of different use scenarios)
Not always a very good measure of suite quality, but “better than nothing”
We “open the box” in white box testing partly in order to look at (and design tests to achieve) coverage
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Coverage Literature of software testing is primarily concerned with
various notions of coverage Ammann and Offutt identify four basic kinds of coverage:
Graph coverage
Logic coverage
Input space partitioning
Syntax-based coverage
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Graph Coverage
Cover all the nodes, edges, or paths of some graph related to the program
Examples:• Statement coverage• Branch coverage• Path coverage• Data flow coverage• Model-based testing coverage• Many more – most common kind of coverage, by far
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Statement/Basic Block Coverage
if (x < y){ y = 0; x = x + 1;}else{ x = y;}
4
1
2 3
x >= yx < y
x = yy = 0
x = x + 1
if (x < y){ y = 0; x = x + 1;}
3
1
2x >= y
x < y
y = 0x = x + 1
Statement coverage:Cover every node of thesegraphs
Treat as one node becauseif one statement executesthe other must also execute(code is a basic block)
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Branch Coverage
if (x < y){ y = 0; x = x + 1;}else{ x = y;}
4
1
2 3
x >= yx < y
x = yy = 0
x = x + 1
if (x < y){ y = 0; x = x + 1;}
3
1
2x >= y
x < y
y = 0x = x + 1
Branch coverage vs.statement coverage:Same for if-then-else
But consider this if-thenstructure. For branch coveragecan’t just cover all nodes, butmust cover all edges – get tonode 3 both after 2 and withoutexecuting 2!
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Path Coverage
if (x < y){ y = 0; x = x + 1;}else{ x = y;}
if (x < y){ y = 0; x = x + 1;}
4
1
2 3
x >= yx < y
x = yy = 0
x = x + 1
6
4
5x >= y
x < y
y = 0x = x + 1
How many paths throughthis code are there? Needone test case for each toget path coverage
To get statement and branchcoverage, we only need twotest cases:1 2 4 5 6 and 1 3 4 6
Path coverage needs two more:1 2 4 5 61 3 4 61 2 4 61 3 4 5 6
In general: exponential inthe number of conditional branches!
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Logic Coverage
if (((a>b) || G)) && (x < y)){ y = 0; x = x + 1;}
3
1
2 ((a <= b) && !G) || (x >= y)
((a>b) || G)) && (x < y)
y = 0x = x + 1
if (x < y){ y = 0; x = x + 1;}
What if, instead of:
we have:
Now, branch coverage will guaranteethat we cover all the edges, but doesnot guarantee we will do so for allthe different logical reasons
We want to test the logic of the guardof the if statement
Dr. A
mr
Tala
at
Pro
toco
l En
gin
eering
Testing “for” Coverage
Never seek to improve coverage just for the sake of increasing coverage
Coverage is not the goalFinding failures that expose faults is the goalNo amount of coverage will prove that the
program cannot fail
“Program testing can be used to show the presence of bugs, but never to show their absence!” – E. Dijkstra, Notes On Structured Programming