PPT 4: Reproducing the ProblemCEN 4072 - Software Testing

First Things First: Reproduce the Problem

Reproducing the problem is important for two reasons

Observe the problem: if unable to reproduce the problem the error cannot be deduced

Check whether the problem is fixed: Recreate the original scenario with the design change to demonstrate

Initial Debugging Process

Step 1 - Reproduce the problem environment

The Setting in which the problem originally occurred

Step 2 - Reproduce the problem history

Prior steps to create the problem

First Things First: Reproduce the Problem

Usually debugging in the problem environment leads to the best chance of reproducing the problem, however may not be possible due to:

Privacy: the user may not wish for others to operate their machines

Ease of development: User may not have a complete development environment (diagnostic software)

Cost of maintenance: User may require that the machine is operational and in their possession

Travel cost: Cost of either travelling to the machine or being sent the machine

Risk of Experiment: Debugging involves experimenting and may cause damage to machine

Iterative Reproduction

1. Attempt to reproduce the problem in your environment, if the problem occurs you are lucky

2. If initial problem does not occur then add more circumstances from the problems environment, (Config Files, Drivers, Hardware)

Start with circumstances

That are likely to cause problems

Easy to change

3. Adopt additional circumstances until

Problem is reproduced

Your environment is identical to problem environment

Iterative Reproduction

If the development and problem environment are identical there exists two alternatives:

The problem could not occur in the problem environment and the problem report may be incomplete or wrong

The Problem report is accurate but there still exists a difference between environments that was unstated by user

Iterative Reproduction

If reproducing environment was insufficient then reproducing the execution may lead to success

The execution of a program is determined by code (assumed constant) and user input

To reproduce user input, programmer must

Observe the input

Control the input

Input may be Deterministic or Nondeterministic

Deterministic: input controllable by the user

Nondeterministic: input not controlled by user

Program InputsInputs for problem reproduction

Data: stored in files and databases, easily reproducible

User Interaction: Can differ in minor details, difference may be important

Communication: data between threads, processes, or machines

Time: Can influence program execution

Randomness: By definition make every execution different

Operating System: Provides services beyond the above and can influence program execution

Schedules: Normally doesn’t interfere but if the program has a defect it may

Debugging Tools: Interferes with programs execution may uncover but also mask problems

Physics: Unintended electrical discharges and radiation may affect program execution.

Data

Reproducing Data: data stored in files/databases, seldom an issue

Only three issues to be aware of:

Get all the data you need: reproduce all data the application accesses and all data under user’s control.

Includes configuration data such as registries or configuration files

Get only the data you need: Some programs use enormous amounts of data, so simplifying input data may be prudent

Take care of privacy issues: User’s data may contain private or confidential information.

User Interaction

Capturing interaction: Program operate normally, interacting environment but all inputs from environment are recorded to a script before program uses it

Replaying interaction: Program executes under control of the tool

Tool redirects program input so the program no longer receives input from the environment

Communication

Record since last checkpoint

Not all data is required. If a program goes into a reproducible state while operating only capture during that state

Record last transaction

Failures are more likely caused by later communications than earlier. Suffices to capture just the latest communications.

RandomnessProgram behavior in different runs

Individual runs may be impossible

Randomness must be made controlled and reproducible

Check using random input

If utilizing pseudo random numbers, capture initial seed of the random number generator

If utilizing Cryptographic, then record all sources of randomness

User Input

Network Events

Thermal Noise

Audio

Operating System

Record and replay OS interactions

Rather than setting up a layer between a program and the OS, isolate arbitrary program units from their environment

Traced interaction

Works by diverting the calls to the OS to wrapper functions that log the incoming and outgoing data

Challenges of replaying traced interactions

The primary problem is that every interaction must be replayed which eats a lot of memory

SchedulesThread changes

Debugging here is potentially the most difficult

When multiple threads attempt to change the same file sourcing the error may be impossible

To solve accessing threads must “lock” the file, thus protecting it.

Or check last update time before accessing and double checking before saving

Reproducing schedules

Like reproducing communications, performance may become an issue by reproducing all inputs

A program should behave identically so one may attempt to uncover difference in execution as induced by schedule differences

Debugging Tools

Code may fail outside the debugger only

Simply observing or examining the problem may cause the problem to disappear, or be replaced by another otherwise known as a Heisenbug

Analogy to Heisenberg’s uncertainty principle

May be caused by uninitialized variables

Debugging tools may also introduce timing issues

Extended run time by routing commands to the debugging environment

Bugs within the debugging tools may cause additional issues

Debugging Tools

Types of bugs:

Bohr bug: Repeatable bug - one that manifests reliably under unknown well-defined set of conditions

Heisenbug: Bug that disappears or alters its behavior when one attempts to probe or isolate it

Mandelbug: Bug whose underlying cause are so complex and obscure as to make its behavior appear chaotic or non-deterministic.

Schroedinbug: Design or implementation bug that does not manifest until someone reading source or using the program unusually notices it never should have worked

In which the program stops working until everything is fixed

Isolating UnitsThe general premise is not to reproduce the execution of the entire program but rather the execution of a

specific unit

May be easier than controlling entire program

However only this unit will provide information

Capture and replay units

Create mock object that would do nothing but simulate the original object using recorded interactions.

Unit control layer

For bottom-level units simply record and replay the interaction at the unit boundary

Isolated Units

Databases: To reproduce problem, record and replay the SQL transactions given by an application - rather than executing the entire program

Compilers: To reproduce problem, record and restore the intermediate data structures - rather than the entire front end

Networking: To reproduce problem, record and replay the basic communication calls as issued by the application - rather than the entire application.

Log as a Program

Aim is to create a control layer that logs all input and output of a mapping from string to integers

Simple way of creating such a map is to create a log as a stand-alone program file

These log files can be compiled and executed, thus reproducing the interaction of a map object and its environment

Logging Interaction with the OS

clog: a C++ buffered output stream generally used for error / debug messages

The code example on the right is a simple program attempting to open a file and if it fails displays an Error message

Other Interaction Types

Variables: Some units provide variables that applications can access to alter unit’s behavior, or retrieve query results

These implicit communications via variables must be controlled and require a lot of work

Unless capture/replay mock tools are utilized

Time: Some units depend on a specific amount of time to elapse between function calls

Record and replay these time intervals for debugging

Other Units: Some units depend on other unit, which may also be controlled by the application

Be sure to capture and restore the states of these units

Breaking dependency of the units may be needed

ToolsWINRUNNER

Records and replays user input for Windows and UNIX systems, http://www.mercuryinteractive.com/

CHECKPOINT

Allow the “freezing” of a running process so it can be resumed later

Even on a different machine, http://www.checkpointing.org/

ANDROID

Open-source GUI testing tool, http://www.wildopensource.com/larry-projects/android.html

REVIRT

Records and replays interaction of entire machine, http://www.eecs.umich.edu/CoVirt/