Debugging and Testing

(Chapter 8)

Testing

● Testing is the process of running a program with the intent of finding bugs

● Requires running the program on sample data● Might require writing extra code to emulate

missing parts of the system● Requires the programmer to be "egoless"● Often performed by programmers other than the

developers

Types of Testing

● Static: Done without actually executing program– Code inspections– Walkthroughs

● Dynamic: Done by executing program or its parts– Module or unit testing– Integration testing– System testing

Module or Unit Testing

● Tests individual classes or small sets of classes● Generally done by the programmer responsible

for the classes● Bottom-up process● May require the use of a throw-away test driver

program

Test Driver for FarmerStateInfoint main() {

Side farmer = EAST; Side wolf = EAST; Side goat = WEST; Side cabbage = WEST;

FarmerState s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();

cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test wrong" << endl; else cout << "Safety test OK" << endl;

farmer = WEST; s = new FarmerStateInfo(farmer, wolf, goat, cabbage); s->display();

cout << "Testing safety:" << endl; if ( s->isSafe() ) cout << "Safety test OK" << endl; else cout << "Safety test wrong" << endl;}

Test Driver Output

67% farmer ||F ||WG|| C||

Testing safety:Safety test OKF|| ||WG|| C||

Testing safety:Safety test OK68%

Notes on the Output

● What gets tested:– The constructor– The display() method– Safety of the cabbage, both when it's vulnerable and

when it's not– The safety of the goat when it's not vulnerable

● What does not get tested:– The safety of the goat when it is vulnerable

● Note: the isSafe() method had to be made temporarily public

Testing State-Changing Methodsint main() {

FarmerState s = new FarmerStateInfo(WEST, WEST, WEST, WEST); s->display();

State newst = FarmerStateInfo::self(s); cout << "Action farmer-takes-self " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;

newst = FarmerStateInfo::wolf(s); cout << "Action farmer-takes-wolf " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl; . . .

newst = FarmerStateInfo::cabbage(s); cout << "Action farmer-takes-cabbage " << endl; if ( newst != NULL ) newst->display(); else cout << "not allowed" << endl << endl;}

State-Changing Test Output

69% farmerF|| W|| G|| C||

Action farmer-takes-self not allowed

Action farmer-takes-wolf not allowed

Action farmer-takes-goat ||FW|| ||GC||

Action farmer-takes-cabbage not allowed

70%

Testing FarmerProblemInfo and State Equality

int main() {

Problem p = new FarmerProblemInfo(); State start = p->getStartState(); State final = p->getFinalState();

cout << "Testing state display:" << endl;

start->display(); final->display();

cout << "Testing state equality:" << endl;

if ( start->equals(final) ) cout << "Equality check wrong" << endl; else cout << "Equality check OK" << endl;

if ( start->equals(start) ) cout << "Equality check OK" << endl; else cout << "Equality check wrong" << endl;

}

Equality Test Output71% farmerTesting state display:F|| W|| G|| C||

||F ||W ||G ||C

Testing state equality:Equality check OKEquality check OK72%

Note that the test driver could be used for otherproblems just by changing the constructor call.

Testing expand()

int main() {

Problem p = new FarmerProblemInfo(); State start = p->getStartState(); start->display();

Item testItem = new ItemInfo(start, NULL, NULL, 0, 0); ItemArray children = testItem->expand(p); cout << "Num children: " << testItem->getNumChildren() << endl;

for (Integer i = 0; i < testItem->getNumChildren(); i++) { State nextState = children[i]->getState(); nextState->display(); }

}

expand() Test Output

73% farmerF|| W|| G|| C||

Num children: 1 ||FW|| ||GC||

74%

Integration Testing

● Putting together the results of module testing● Top-down approach● Example: testing the SolverInfo class tests

the integration of ProblemInfo and QueueInfo

● Often includes a primitive graphical user interface (GUI) so that:– system has look and feel of final product– separate drivers not needed

● Might require stubs or dummy routines for code to be added later

System Testing● Attempts to find problems in a complete program● Done when

– program is first completed, and– as the program evolves (maintenance testing)

● In industry, often done by an independent group of programmers

● Requires creation of a test suite that:– is used over and over as program evolves– attempts to identify all combinations of inputs and

actions that could cause program to fail

Testing and Proof of Correctness

● It is not possible to test a nontrivial system with all possible inputs and outputs

● So testing cannot formally prove that a system is correct, only give some level of confidence in it

● Formal proofs of dynamic program correctness are not possible either, because executing programs are subject to physical influences

● Formal proofs of abstract algorithm correctness are possible though difficult, and require machine assistance

Debugging

● The process of fixing problems (bugs) that testing discovers

● Debugging has two parts:– finding the cause of a bug (can be 95% of debugging)– fixing the bug

● A good programmer becomes adept at using a run-time debugger

Types of Program Bugs

● Syntactic bugs not caught by the compiler

Example:for (... ; ... ; ...);{

...

}● Design bugs: easy to find, can be difficult to fix● Logic bugs: can be difficult to find, easy to fix

Types of Bugs (cont'd)

● Interface bugs: – When assumptions made by a method caller are

different than those of the callee. – Can be difficult to find, easy to fix

● Memory bugs:– Freeing or overwriting storage that is later used– Especially difficult to find– Example:

IntArray A; // = new Integer[4]; ...A[0] = 10;A[1] = 20: ...

Fixing Bugs● Fixing might entail correcting a single line of

code● Fixing might entail redesign, recoding, and

further testing:– For example, incorrect circularity check in expand()

● Fixing a bug might cause other bugs to appear● It has been observed that code in which a bug has

been found is more likely to still contain bugs

Defensive Programming

● The easiest way to do debugging is to not have bugs

● Failing that, one should strive for locality:

Ensure that bugs that are identified are local problems found where and when they occur

● Locality is the goal of defensive programming● Defensive programming begins with defensive

design

Defensive Design

● First goal: Simplicity of Design

Example: Take the time and trouble to make a repeated piece of code a method, even if small

● Also: Simplicity of Class Interfaces

Example: refactoring the abstract StateInfo and ProblemInfo classes

● Another goal: Program for Errors– Anticipate exceptions even when they are not

expected– Example: add(Item) and remove() should check

for fullness and emptiness even though currently all callers check first

Program Robustness

void FrontQueueInfo::add(Item item) { front++; items[front] = item; numItemsAdded++;}

Consider this add method for FrontQueueInfo:

If add is called and the queue is full, a probable segmentation fault will occur.

As written, this code is brittle.

A robust program will not crash and burn whensomething unexpected happens.

Relying On the Caller

FrontQueue q = new FrontQueue(n); . . .if ( !q->full() ) {

q->add(item);}else {

<deal with full queue>}

However, as the program evolves, it is possible thata call will be added that is not this protective.

One approach: try to ensure that every call protectsitself:

Defensive Coding Using assert

add itself could have some kind of defense againstan incorrect call:

void FrontQueueInfo::add(Item item) {assert( !full() );front++;items[front] = item;numItemsAdded++;

}

If the queue is full, this will result in a helpfulmessage and a program halt.

Although this makes it easier to debug, the programis still brittle.

Defensive Coding Through Exception Handling

An exception is an unexpected error condition:

1)Array index out of bounds

2)Divide by zero

3)Heap memory exhausted

4)Keyboard interrupt signal

5)Abort signal

6)Conditions specific to application

C++ Approaches to Exception Handling

1 Try to guarantee that an exception condition does not exist through an assertion, and generate an unrecoverable error if the assertion is false.

2 Deal with an exception condition through the use of a handler function that is called when a system-defined exception signal is raised.

3 Deal with general exception conditions by unwinding the stack with catches and throws.

Defensive Coding Example: Safe Arrays

expected output: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9

actual output:

0 1 2 3 4 5 6 7 0 1 0 1 2 3 4 5 6 7 8 9

problem: accessing array elements that are out of bounds

typedef int Integer;typedef Integer * IntegerArray;

main () { IntegerArray a = new Integer[5]; IntegerArray b = new Integer[5]; for (Integer i = 0; i < 10; i++) a[i] = i; for (Integer i = 0; i < 10; i++) b[i] = i; for (Integer i = 0; i < 10; i++) cout << a[i] << " "; cout << endl; for (Integer i = 0; i < 10; i++) cout << b[i] << " "; cout << endl;}

A Safe Array VectorInfo Class#include <assert.h>

typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;

class VectorInfo {private: IntegerArray p; Integer size;public: VectorInfo(); VectorInfo(Integer n); ~VectorInfo(); Integer& element(Integer i);};

Notes On VectorInfo

● One class attribute is a dynamic array● One class attribute is the maximum array size● One constructor will implement a default array

size● The element method will check for index out

of bounds● assert.h is included

Exception Handling Using assert

VectorInfo::VectorInfo() { size = 10; p = new Integer[size];}

VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}

Integer& VectorInfo::element(Integer i) { assert (i >= 0 && i < size); // line 40 return (p[i]);}

Main Programmain () { Vector a = new VectorInfo(5); Vector b = new VectorInfo(5); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl;}

Main Program Output

VectorInfo2.cc:40: failed assertion `i >= 0 && i < size'Abort

Although this code aborts, it is preferable becauseit gives the programmer useful information.

From the point of view of the user, it is still brittle.

Exception Handling with signal.h● The signal.h file provides a standard mechanism for

handling system-defined exceptions. Some:#define SIGHUP 1 /* hangup */

#define SIGINT 2 /* interrupt */

#define SIGILL 4 /* illegal instruction */

#define SIGFPE 8 /* floating point exception */

#define SIGBUS 10 /* bus error */

#define SIGSEGV 11 /* segmentation violation */

A programmer can arrange to have a handler function called when an exception arises using:

signal(signal, handler);

Signal Handler Function● Returns void and takes an integer (signal number) as

argument:– void handler(Integer signal);

● If part of a class, must be static ● Installed by associating it with a signal:

– void signal(Integer signal, void (*handler) (Integer))

● Automatically invoked when an exception of the associate signal type occurs

● Can be intentionally invoked by explicitly raising the associated signal:– void raise(Integer signal);

Signal Example: Keyboard InterruptMain Program

#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype

main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;

signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''

while (i < j) // interrupt with ctl-C during this loop ++i;

cout << " HIT " << j/1000000 << " MILLION" << endl; }

Keyboard Interrupt Handler

void cntrl_c_handler(int sig){ char c;

cout << "KEYBOARD INTERRUPT"; cout << "\ntype y to continue: "; cin >> c; if (c != 'y') exit(0); signal(SIGINT, cntrl_c_handler); // reset "trap" // and return}

Signal Example Output

6% testCOUNT TO J MILLION, Enter j: 100

[control-C from keyboard]

KEYBOARD INTERRUPTtype y to continue: y

[control-C from keyboard]

KEYBOARD INTERRUPTtype y to continue: yHIT 100 MILLION7%

Raising An Exception Under Program Control

#include <signal.h> ...void cntrl_c_handler(int sig); // handler prototype

main() { int i = 0, j; cout << "COUNT TO J MILLION, Enter j: "; cin >> j; j *= 1000000;

signal(SIGINT, cntrl_c_handler); // set interrupt ``trap''

while (i < j) // interrupt with ctl-C during this loop ++i;

if (i % 1000000 == 0) // generate interrupt after raise(SIGINT); // each million

cout << " HIT " << j/1000000 << " MILLION" << endl; }

Signal Example Output

9% testCOUNT TO J MILLION, Enter j: 4KEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: yKEYBOARD INTERRUPTtype y to continue: n530%

Note that the program does not continue to completion.

User-Defined Signals

. . .#define SIGUSR1 16 /* user defined signal 1 */#define SIGUSR2 17 /* user defined signal 2 */ . . .

Two signal codes are designed to be raised by programmers in application-specific circumstances:

User-Defined Signal Example

Recall the safe array constructor:

If the heap is exhausted (p==0), this code will abort.

Suppose the programmer can do some garbagecollection to reclaim parts of the heap.

VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; assert (p != 0); // in case heap used up}

User-Defined Signal Example (cont'd)#define SIGHEAP SIGUSR1

#include <assert.h>

typedef int Integer;typedef Integer * IntegerArray;typedef class VectorInfo * Vector;

class VectorInfo {private: IntegerArray p; Integer size;public:

VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer

sig); };

User-Defined Signal Example (cont'd)

VectorInfo::VectorInfo(int n) { assert ( n >= 1 ); size = n; p = new Integer[size]; if (p == 0) { // invoke handler to do

raise(SIGHEAP); // garbage collection p = new int[size]; // try again to create

array }}

main () {signal(SIGHEAP, VectorInfo::vectorHeapHandler);vect a(5);vect b(5); . . .

}

void VectorInfo::vectorHeapHandler(int sig) { // possible action to reclaim heap storage}

Limitations of Signal Handling for C++

It would be nice to handle the other VectorInfo exceptions in this way, for example:

#define SIGHEAP SIGUSR1#define SIGSIZE SIGUSR2

class VectorInfo {private: IntegerArray p; Integer size;public:

VectorInfo();VectorInfo(Integer n);~VectorInfo();Integer& element(Integer i);static void vectorHeapHandler(Integer

sig);static void vectorSizeHandler(Integer

sig); };

Limitations of Signal Handling for C++ (cont'd)

VectorInfo::VectorInfo(int n) { if ( n < 1 ) {

raise(SIGSIZE); // bad vector size signal}

size = n; p = new Integer[size]; if (p == 0) {

raise(SIGHEAP); p = new int[size];

}}

Limitations (cont'd)

main () {signal(SIGHEAP,

VectorInfo::vectorHeapHandler);signal(SIGSIZE,

VectorInfo::vectorSizeHandler);vect a(5);vect b(5); . . .

}

void VectorInfo::vectorSizeHandler(int sig) { cout << "Size error. Going with default." size = 10; p = new int[size];}Unfortunately, this will not work since vectorSizeHandler must be static. It therefore has no access to size and p.

Exception Handling with catch/throw

● A more sophisticated and general form of exception handling

● Not intended to handle asynchronous exceptions defined in signal.h

● Works by causing dynamic, non-local exit from runtime stack frames ("unwinding" the stack)

Normal Runtime Stack Handling

main() {. . .foo();. . .

}

void foo() {. . .bar();. . .

}

void bar() {. . .wowie();. . .

}

void wowie() {. . .zowie();. . .

}

void zowie() {. . .. . .

}

call

return

call

return

call

return

call

return

Normal Runtime Stack Handling (cont'd)

In zowie, the runtime stack looks like:

zowiewowie

barfoo

main

Each normal return causes the stack to be popped:

zowiewowie

barfoo

main

wowiebarfoo

main

barfoo

mainfoo

main main

Non-Local Exits Using catch/throw

● Non-local exits from stack frames are caused by throwing an exception to a handler (a catch)

● The throw and the catch can be widely separated on the stack

● All intervening stack frames are "unwound" (removed from stack along with all automatic values)

● Control does not pass to the code after a call● Control ends up at the catch, which should take

appropriate action

Non-Local Exits Using catch/throw (cont'd)

zowiewowie

barfoo

main main

main () { try {

foo(); } catch(int n) { . . . }}

. . .

void zowie () { int i; . . . throw i; . . .}

● The try block establishes the context for any throws done within its dynamic extent

● The try block is immediately followed by handlers which catch a thrown exception

● There can be more than one handler which differ according to parameter type signature

code stack

Using catch/throw in VectorInfo

VectorInfo::VectorInfo(Integer n){ if ( n < 1 ) throw (n); size = n; p = new Integer[size]; if ( p == 0 ) throw ("Free Store Exhausted");}

Integer& VectorInfo::element(Integer i){ if (i < 0 || i >= size) throw ("Vector Index Out Of Bounds"); return (p[i]);}

VectorInfo Main Program

void process(Integer m) { try { Vector a = new VectorInfo(m); Vector b = new VectorInfo(m); for (int i = 0; i < 10; i++) a->element(i) = i; for (int i = 0; i < 10; i++) b->element(i) = i; for (int i = 0; i < 10; i++) cout << a->element(i) << " "; cout << endl; for (int i = 0; i < 10; i++) cout << b->element(i) << " "; cout << endl; } ...

main (Integer argc, StringArray argv) { Integer n = atoi(argv[1]); process(n);}

VectorInfo Main Program (cont'd)

. . . catch (Integer n) { cerr << "Size error. Going with default." << endl; process(10); } catch (ConstString message) { cerr << message << endl; exit(0); }} 7% test 10

0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9

8% test 0Size error. Going with default.0 1 2 3 4 5 6 7 8 90 1 2 3 4 5 6 7 8 9

9% test 5Vector Index Out Of Bounds

Output:

catch/throw in the Queue Command Interpreter

void CommandInterpreterInfo::execute() { cin >> cmd; try { while ( !cin.eof() ) { count++; process(); cin >> cmd; } } catch (ConstString message) { cout << message << endl; } cout << count << " commands processed." << endl; free();}

Command Interpreter (cont'd)

void CommandInterpreterInfo::process() { if (cmd == '#') { processComment(); count--; } else if (count == 1) { processCreate(); } else { processOther(); }}

Command Interpreter (cont'd)

void CommandInterpreterInfo::processCreate() { if (cmd != 'c') throw ("First command must be a create..."); cin >> qtype; checkEOF(); if ( (qtype == 'f') || (qtype == 'r') || (qtype == 'p') ) { if (qtype == 'p') { cin >> pqtype; checkEOF(); if ( !(pqtype == '>' ) && !(pqtype == '<' ) ) { throw ("Priority queue type must be > or <."); } } cin >> qsize; checkEOF(); switch ( qtype ) { case 'f': q = new FrontQueueInfo(qsize); ... case 'r': q = new RearQueueInfo(qsize+1); ... case 'p': switch ( pqtype ) { case '>': q = new MaxPriorityQueueInfo(qsize); ... case '<': q = new MinPriorityQueueInfo(qsize); ... } } } else throw ("Queue type must be f, r, or p.");}

Command Interpreter (cont'd)

void CommandInterpreterInfo::checkEOF() { if ( cin.eof() ) { throw ("Unexpected end of file"); }}

execute

process

processCreate

checkEOF

RuntimeStack:

try/catch here

throw from here

throw from here

Philosophy of Error Recovery Using Exception Handling

● Exception handling is about error recovery and secondarily about transfer of control

● Undisciplined transfer of control leads to chaos (like the days before structured programming)

● In most cases programming that raises exceptions should print a diagnostic message and gracefully terminate

● Heroic attempts at repair are legitimate in real-time processing and fault-tolerant computing