Software Verification and Validation

Objectives

Testing allegiance to customer not development team

Test ALL documents Intent: Find Defects

Good test = test has high probability of finding an error

Successful test = test that uncovers an error Prevent defect migration (apply early in life-

cycle) Develop and Use Test Tools Trained, skilled people in testing

People

Creatively destructive Hunt for errors (not people who create errors) Trained for testing 25% of development time spent on testing,

but teachers spend 5% teaching it

Software Verification and Validation

Verification – Are we building the product right? Set of tasks that ensure SW correctly implements a

specific function

Validation – Are we building the right product? Different set of tasks that ensure SW built is

traceable to customer requirements

Methods for Verification

Formal Proof Inspections Walkthrough BuddyCheck

Formal Formal Informal Very informal

Presenter NONE Not author Anyone None

# people Team 3-6 Larger #s 1 or 2

Preparation Yes Yes Presenter NONE

Data/report Yes…a proof yes ? No

Advantages Very effective Effective Familiarizes

Many

Inexpensive

disadvantages Requires trained mathematicians

Short term cost Fewer errors found Fewer errors found

Verfication: Correctness Proofs

Testing uncovers errors, cannot prove program correctness

Manual correctness proofs Mathematical induction and predicate calculus Feasible only on small programs Proofs can contain errors

Automated Correctness proofs MacroCompiler produces symbolic representation

of SW Predicate calculus and AI theory Limited to types of applications

Strategies and Techniques for Validation

Well-defined activities Low-level: unit testing, integration test High-level: usability, functional test, system test,

acceptance test Regression testing – test unchanged code AGAIN

Testware Black box – derived from requirements and

functional specs/no knowledge of structure or code

White box – based on internal code

Test Information Flow

EvalTesting

Reliability

DebugSWConfig (specs, code)

Test results

Test Config (plan,cases) expected results

errors

Expected results

corrections

Predicted reliability

TESTING STRATEGIES

Software Testing Steps

Unit Testing

Evaluate Module interface

• # of input parms = args, match, order, input-only args altered?, global variable definitions consistent, constraints passed (ex. maxsize of array)

Logical data structure• Improper or inconsistent declaration, erroneous initialization, incorrect

variable names, inconsistent data types, underflow, overflow, address exceptions, global data

File Structure• File attributes correct, OPEN statements correct, format spec matches

I/O statement, buffer size=record size, files opened before use, EOF handled, I/O errors handled, textual errors in output

White box and Coverage Most common errors (computations)

• Arithmetic precedence, mixed mode ops, incorrect initialization, precision inaccuracy, symbolic rep of expression

More Unit Testing

Comparisons• Of different data types, logical operators or precedence NOT,

expectation of equality when precision error, improper or non-existent loop termination

Antibugging• Cleanly terminate processing ore reroute when error occurs

(incorporated, but not usually tested)

Error Handling• Intelligible description, provide enough info, error noted = error

encountered, error handling BEFORE system intervention

Integration Testing

Data can be lost across interface One module can have an inadvertent effect

on another Accumulating imprecision Global variables

Testing at Different Levels

Bottom-up Integration Uses Drivers at many

different levels Disadv:interface

problems appear later in process

Top-down Integration Uses stubs Test main logic first,

add modules, retest Disadv: planned lower-

levels may be impossible to write

Still need entire package validated

Top-down Approach

Bottom-up Approach

Validation Testing

Uses black box tests to demonstrate conformity to requirements

Deviations can rarely be fixed prior to scheduled completion

Configuration review of all deliverables for support of maintenance

Alpha – at developer’s site by customer Beta – at customer’s site -> report back to

developer

System Testing

Incorporated into larger computer-based system Recovery tests within time – cause system to

fail and verify recovery performed Security – testing plays role of person who

breaks in, attacks system, penetrates database

Stress tests – abnormal frequency, volume, quantities

Performance tests – timings/resource utilizations

TESTING TECHNIQUES

Black Box Testing Interface Testing

Unit interfaces and I/O interfaces

Equivalence Partitioning Input class: generate 1 test point for each input class Output class – generate 1 test point that gives result in each output class Subdivide into subclasses (middle of range and 2 extremes) Error Handling Routines/Exception output conditions (input beyond

accepted range) Boundary value analysis

The boundaries of a range Functional Testing

functions w/I certain math classes can be distinguished by their values on a small # of points

F(x) = y, if x > 0; y-1, if x <= 0 Functions of more than one variable

Random inputs Cause-effect graphing testing Comparison testing

White Box Testing

Based on program structure Coverage metrics for thoroughness

Statement coverage Branch coverage (each T/F) Path coverage (every combination of T/Fs)

• Impractical – too many paths• Eliminate infeasible paths

If y < 0 then x = y –1

else x = y+1;

if x > 0 then … then-then infeasible

• Missing paths (“special cases”) – can involve 1 single input data point (if y = 0)

More…Path coverage

• Coincidental correctnessRead x;

y = x + 2; vs. y = x * 2;

Write y;

Can’t distinguish if input is 2 => 1 point per path is insufficient

Paths

Loop <= 20

104 possible paths

At 1 task/msec = 3170 years

More White Box Testing

Data Flow Coverage Mutation Analysis

Create mutant programs See if mutant programs give identical output for each test;

• If yes, mutant is live• If no, mutant is killed

Display set of live mutants, if all killed, confidence in Test Examples:

• Replace 1 constant or variable with another• Replace one arithmetic operator with another• Similarly, relational and logical operators• Delete a statement• Increment change

Tools

Static Analyzers No actual inputs,

compilers, check syntax, unreachable code, undefined references

Code Auditors Assertion Processors Test File/Data

Generation Test Verifier

Test Harnesses Output Comparators Simulators Data Flow Analyzers Symbolic Execution

Systems (verifiers)

Reliability

Based on error rate Based on internal characteristics of program

(complexity, #operands, #operators) Seed SW with known errors and evaluate #seeded

errors detected vs. actual errors detected –this evaluates power of tests