Making Your Program FailCEN 4072

Before the Bugs

Why would a software development team release their work with known bugs? They wouldn’t; they would find all bugs via testing.

Theoretically, all bugs should be found by testing a program before it is released.

Testing is the process of executing a program with the intent of producing some problem. Once a problem is found, the same process of fixing the problem is taken as if the problem was reported by the user.

Different Types of Tests

A developer either tests for a specific problem, or tests for a yet unknown problem.

A specific problem has already been identified. It may have been identified by the developer or by a user, in which case a problem report should have been made.

A test for yet unknown problems is a little more difficult because the developer doesn’t necessarily know what they are looking for. How to find as many unknown bugs as possible before release is a heavily researched topic.

Testing for Debugging

The debugging process of a known problem is carefully laid out in precise steps:

1. Reproduce

2. Simplify

3. Observe

4. Validate

5. Regress

Reproduce

The problem must be reproducible or else the developer won’t be able to test the problem.

A test must be created to reproduce the problem.

Simplify

Once the problem is reproducible, run the test several times to identify similarities and differences of the output.

After you rerun the test multiple times, attempt to simplify the problem.

Observe

Once you have simplified the problem to a more precise problem, rerun the test again.

“Look under the hood” and determine exactly what the problem is.

Validate

After you have observed the specific problem, the developers fix it.

At this point, rerun the test again. You would expect the program to work properly now with no problems.

Verify that the fix has been successful.

Regress

At this point, your software has been released to your customer(s).

Some time later, you are ready to release an updated version of your software.

All previous tests should be rerun on the new software to ensure none of the previous problems occur again.

What might make your code fail?

Invalid Input: if your program asks the user to enter an integer amount, and user enters a decimal, will the code fail?

Code Changes: maybe a loop sometimes leads to an out of bounds access in an array. A code change may be needed.

Thread Schedules: some programs can not execute concurrently. Think about when you try and update a program and your OS informs you that you need to close a certain program to proceed.

Program States: a program that is running may request I/O and then enter an invalid state, causing future failure.

What could go wrong? When trying to automate, or making a test easily repeatable,

one might run into some problems. Lets consider a sample test. Our problem is Mozilla crashes

on the following instructions:

This is an example of a simple problem report.

What could go wrong?

An automated test would include opening Mozilla browser, “click” in the address bar, enter the URL, etc.

What if the machine is slow and the browser doesn’t open by the time the automated test tries to click in the address bar? We have run into a synchronization problem.

What if the problem only occurs when running Mozilla on a Windows OS and doesn’t occur on the developer’s machine with Linux OS? We have experienced a portability problem.

Different Interaction Layers

Presentation Layer handles the interaction with the user.

Functionality Layer encapsulates the actual functionality of the program independent of a specific presentation.

Unit Layer splits the functionality across multiple units, all cooperating together to produce a greater result.

Solid lines represent automated test/user execution. Dotted lines represent the result. Notice how users only interact with the presentation layer.

Testing the Presentation Layer

The first area to test seems quite obvious, the presentation layer. This is the layer the user will interact with, so lets start here.

Long story short, test different inputs and observe the various outputs. This can be done at different abstraction levels.

Low-Level Interaction High-Level Interaction System-Level Interaction

Low-Level Interaction The lowest abstraction level consists entirely of simple

expressions: key strokes and mouse events. To automate the process, the process of events can be

captured and replayed to recreate several tests. As one can imagine, any minor change to the user interface

can break the automated test.

• This is an example ANDROID script of a recorded low-level test for our previous example of launching Mozilla browser, opening URL dialog box, clicking in the text box, and typing the URL.

Higher-Level Interaction Higher abstraction level

controls the system not by coordinates, but by graphical commands.

APPLESCRIPT is displayed on the right. It was designed to be readable to the user. Can you figure out what is happening easily?

High-level interaction doesn’t rely on coordinates, so this testing style is more robust against user interface position changes.

System-Level Interaction

To ensure control, a virtual machine can be introduced to emulate the entire machine.

The software of the virtual machine emulates the entire system, hardware and software. The script for the virtual machine FAUmachine below shows how hardware is emulated.

Different Script Languages ANDROID (low-level) records a stream of mouse clicks and

keyboard interactions. This method is fragile; the method is dependent on coordinates and can break easily if the user interface changes.

APPLESCRIPT (high-level) moves away from coordinate dependency and relies on actual graphical user controls. This method is more portable and robust.

FAUmachine (system-level) is a script language for a virtual machine. A virtual machine can emulate an entire machine. Therefore, it is comprised of commands for both software and hardware.

When to run other tests?

If the program being tested has an elaborate GUI, most likely there is a lot of stuff happening under the hood that needs to be tested on a different level.

Typically, we should only use presentation-layer testing if the problem occurs in the presentation level, if the presentation layer is noncomplex, or if there is no alternative.

The more user-friendly an interface is, the less computer-friendly it is.

Testing the Functionality Layer

The main advantage of testing from the functionality layer versus presentation layer is that results can be easily accessed and evaluated.

Instead of ever using the UI for testing, it is more practical to design your program with an interface that is designed for automation.

• The script code to the right is APPLESCRIPT commands to open Safari and go to a specified URL.

Testing the Individual Units

A unit is any subprogram, function, library, module, structure, class, etc.

Unit Tests, a test that covers a single unit, must be run on each unit to verify proper functionality.

Nowadays, unit tests are automated using a testing framework, that is, they run with no user interaction.

For each unit, the testing framework will set up an environment for the unit to run, tests the unit with all possible input combinations, verifies if the outcome is as expected, and then breaks down the environment it created.

Running Unit Tests

Unit tests can not use scripts, so different testing frameworks are available. We will look at JUNIT example to test URLs.

First of all, the framework must be able to setup and close the unit. This is done via the setUp() and tearDown() methods, respectfully.

askigor_url is a string variable that contains our URL.

Running Unit Tests

Each test you wish to run on a unit is a different method in JUNIT.

For our URL example, we can make 4 tests for the 4 different parts of the URL: protocol, host, path, and query.

JUNIT Graphical User Interface

The left box shows a passing test on URLs.

The right box shows the results of a test with 1 error.

pyUNIT Unit Testing Example Just like JUNIT,

the Python Unit Testing also has to setup and close units, as well as test them. Here is a simple example that tests simple addition.

Circular Dependency

Our previous example of unit testing URLs is easy because the URL doesn’t depends on anything. Now consider a function that writes a webpage to a file, but first prompts the user for permission to avoid overwriting files.

A unit test of this function runs into a problem when the user prompt comes into play. That is the presentation level, and we are trying to work with the functionality level.

The presentation layer invokes print_to_file() which is dependent on the functionality layer.

The functionality layer invokes confirm_loss() which is dependend on the presentation layer.

Breaking Circular Dependency

One way to avoid these circular dependencies is to “hack” the program into automatically returning true to the user prompt when in testing mode. In this case, testing mode is called “AutomatedPresentation.”

Breaking Circular Dependencies The general idea of breaking circular dependencies is the

dependence inversion principle which highlights importance of depending on abstractions rather than details.

Given component A dependent on component B, to break the dependence:

1. Introduce abstract superclass B’, make B subclass of B’.

2. Set up A such that it depends on B’ rather than B.

3. Introduce alternate subclasses of B’ that can be used with A such that B is no longer required.

Good Debugging Design (MVC)

The most popular design is the model-view-controller architectural pattern.

The model holds the core data and functions that operate on the core data.

There also exists observers that are attached to the model and are notified whenever the core data changes.

The view is responsible for displaying the core data in a specified way.

And finally, the controller handles input events and invokes services on the model.

General Design Rules

All software design methods focus on two major principles to minimize dependencies:

1. High Cohesion is the idea of grouping similar aspects of a project into classes and objects. For example, an ‘employee’ class contains data such as jobTitle and employeeName as well as functions like promote() and updateName().

2. Low Coupling is the idea of keeping unassociated data separate and independent. If two different units do not operate on common data, keep them independent. For example, an ‘employee’ and ‘product’ class should always be separate because employees are not dependent on the product they sell.

Essential Rules for Testing Specify. A program can only be correct as long as its

specifications are defined. Test Early. Run tests from the beginning of development. Test First. Write test classes before designing the unit. Test Often. Continue to frequently test throughout execution. Test Enough. Ensure you have thoroughly tested your

program. Have Others Test. Just like proofreading an essay, code

always looks different in someone else’s eyes. They could possibly find errors or solve errors that you would never think of.

Tools for Automated Debugging

JUNIT available at http://www.junit.org/ APPLESCRIPT available at

http://www.apple.com/applescript/ VBSCRIPT available at http://en.wikipedia.org/wiki/VBScript Other scripting languages like ANDROID, PYTHON, PERL,

TCL, and JAVASCRIPT can possibly be found on the web.

Recap To test for debugging, one must create a reproducible test,

run the test several times to identify the specific problem, validate the revised software, and always retest new software in the future.

There are different levels of abstractness when testing: presentation, functionality, and unit-level tests.

Testing the presentation and functionality level allows the use of scripts like APPLESCRIPT.

Low level scripts depends on coordinates of the UI while high level scripts depend on actual graphical objects.

Unit tests are achieved using testing framework like JUNIT. Use a MVC architecture as well as high cohesion and low

coupling principles to help reduce the number of dependencies and avoid circular dependencies.

Sources Zeller, A. (2009). Why Programs Fail: A Guide to Systematic

Debugging(2nd ed.). Burlington, MA.

http://www.thelinuxdaily.com/2011/11/getting-started-with-python-unit-testing-pyunit/

https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller