telecooperation/rbg technische universität darmstadt copyrighted material; for tud student use only...

Telecooperation/RBG

Technische Universität Darmstadt

Copyrighted material; for TUD student use only

Introduction to Computer Science ITopic 19: Quality Control

Prof. Dr. Max MühlhäuserDr. Guido Rößling

Dr. G. RößlingProf. Dr. M. Mühlhäuser

RBG / Telekooperation©

Introduction to Computer Science I: T19

Outline

2

• Introduction to Quality Control

• Design by Contract

• Structural and Functional Test Methods




Quality Control: Motivation

• This code from java.util.Arrays had a bug for 9 years:

3

public static int binarySearch(int[] a, int key) { int low = 0; int high = a.length - 1; while (low <= high) { int mid = (low + high) / 2; int midVal = a[mid]; if (midVal < key) low = mid + 1; else if (midVal > key) high = mid - 1; else return mid; // key found } return -(low + 1); // key not found.}





• X-rays, Therac-25 (`85)– wrong doses caused by a rarely

occurring condition and due to an overflow in the counter for the positioning of the beam

– Patients were hurt or killed

• Ariane flight 501 (`96) – loss of rocket and satellites

($500,000,000)

• AT&T Telephone network (1990)• Airbus 320 (`93 and 2001)• many, many more...

4

Computer Science Catastrophes




Quality Control

• Constructive approach– Aim: construct error-free

software• e.g., function composition

• Analytic approach

– Aim: Prove that the product is free of errors respectively finding errors, if any exist

– checking the product, e.g. using…

• consistency checks• comparison with specification • validation by customer

5

Two complementary approaches

ok

ok

ok

Definition

Design

Implemen-tation e

rror

rem

oval




Why We Need to Test

• As long as human brains develop software, it will (in general) contain errors– To err… is human

• Errors must be caught before they have detrimental effects

6

Where are the

needles?

Hungry!




How intensively should we analyze?

• How can you search thoroughly?• At which point can you expect to have come

across all ”needles" with a very high probability?

7

Hurry up!?!

Have I got them all?




What is the task of analysis?

• Just detecting errors• Finding the cause and removing the culprit

(debugging) is another activity

8

And now?

There's one left some-

where!





• Errors occur in a wide spectrum:– not dramatic: clients are used to low

standards– undesirable: clients may be lost– intolerable: damage may result

• Depending on the necessity to avoid errors, various methods are used– formal– empirical

9




Different Approaches

• Mathematician: 3 is a prime, 5 is a prime, 7 is a prime; by induction, all odd numbers are prime

• Physicist: 3 OK, 5 OK, 7 OK, 9 measurement error, 11 OK, 13 OK, looks to be true

• Engineer: 3 is a prime, 5 is a prime, 7 is a prime, 9 is a prime, 11 is a prime, …

• Computer Scientist: 112 is a prime, 1012 is a prime, 1112 is a prime, …

10

Hypothesis

"All odd numbers greater than 1 are prime"

Caution!This is a satire.




Quality Control Methods

11

e.g., Java Compiler:check whether variables are initialized and return statements are present

analytic quality control

testing analyzing methods

dynamic test

structural test

functional test

verification static analysisreview

(e.g., inspection)




Quality Control Terminology

• Verification: “are we building the software correctly?”– Proof that a product is correct with respect to the

specification(functional tests as a coarse approximation)

• Validation: “are we building the right software?”– Confirmation that a product runs in a certain target

environment and does what the user wants

12

• A verified program may not be in accordance with a customer's desires

• A validated program need not fulfill the specification




Different Approaches

• Non-systematic "methods" are not acceptable• “Checking here and there" (ad-hoc testing) is not

suitable for gaining sufficient confidence

• Systematic checking (coverage testing) may be sufficient, but typically cannot guarantee the absence of loopholes

• Formal checking (formal verification) may guarantee absolute correctness

13

In reverse order of rigor

Beware of bugs in the above code; I have only proved it correct, not actually tried it.

(Donald Knuth)

Testing can only show the presence of errors, never their absence. (Dijkstra’s Law)




Testing Dilemma

14

If something can go wrong, it will— at the worst possible moment. If nothing can go wrong, it still will. If nothing has gone wrong, you have overlooked something.

Murphy’s Law

Blinn's Thesis: Every working computer graphics programcontains an even number of sign errors




Cost of Testing

15

• Detection rate• increasingly difficult to find

new errors

• Tests / found errors• tests get more elaborate

• Error quality• errors become nastier

• Combination • Cost per found error rises

very quickly• Shall we stop if it gets too

difficult/expensive?

time




Necessity of a Systematic Approach

• In practice, the following (unsatisfactory) reasons are used to stop testing:– time's up; customer wants the product to be

shipped– all existing tests are passed

• This calls for objective criteria to verify whether more tests are needed test methods

• We have already regarded practical testing with JUnit

16




Outline

17







Design by Contract

Recall: • We already have used contracts in Scheme• Define a contract for each function or operation• Parties in the contract: calling class and called

class

• Contract entails benefits and obligations for both parties

18

“If you promise to call op with precond satisfied, then I, in return, promise to deliver a final state in which postcond is satisfied.”




Design by Contract

• A contract consist of:– Precondition: before executing a certain operation– Postcondition: after executing a certain operation (before

exit)

– Internal Invariants: within a certain scope (in a method)– Control Invariants: a certain control flow must (not) be

entered– Class Invariants: always true before and after each

operation• but not necessarily while the object is in transition

– Effect description: describes the semantic of the operation and its (side) effects

19




Design by Contract• Comparing Scheme and Java

– In Scheme, we had to check the argument data types– In Java, we only have to check argument data types in a

few cases • e.g., Object[], where we expect only some subtypes of Object to be contained in the array

• Design by Contract and OOP– Methods of subtypes inherit the contract of the method of

the super-type– Same preconditions and postconditions

• May have stronger preconditions and postconditions– Throw the same exceptions

• May throw more specialized exceptions

20




Assertions• The assert keyword is a basic language support for Design

by Contract in Java since Java 1.4– We can define assertions to constrain:

• the use of the object interface• the internal state of an object

• Assertions have the formassert <booleanExpression> [: <Expression>]– If booleanExpression is true, the program executes normally.

Otherwise, an AssertionError will be thrown.– Expression is an optional expression that will be evaluated and

printed in the error message at runtime.– The AssertionError class is a subtype of Error class. Similar

to RuntimeException, it need not be caught or declared to be thrown by methods

• By default, assertions are not active– They can be activated by java –ea <filename>

21




Using Assertions• assert can be used to express:

– Precondition,– Postconditions,– Internal invariants,– Control invariants,– Class invariants

• Be careful in certain cases:– Do not use assertions for argument checking in public

methods– Do not use assertions to do any work that your

application requires for correct operation • The assert statement is only evaluated if assertions are

activated!– Assertion should not have any side effects on normal

program execution

22




Using Assertions for Preconditions• An exception for preconditions of public methods:

– Do not use assertions for argument checking in public methods Published specification (or contract) must be always obeyed By convention, use an appropriate runtime exception:

usually, this will be an IllegalArgumentException

• Preconditions of private methods:

23

public void setAttr(int anAttr) { if (anAttr > 0 && anAttr <= MAX_VALUE) throw new IllegalArgumentException("illegal value:"+anAttr);

attr = anAttr;}

private void setAttr(int anAttr) { assert anAttr > 0 && anAttr <= MAX_VALUE: "value for attr is not in interval:"+anAttr;

attr = anAttr;}




Using Assertions for Postconditions

• Postconditions of both public and private methods:

24

public String reverse(String oldStr) { String newStr = "";

// move each char of old to new string by reverse order

assert oldStr.equals("") : "oldStr must be empty"; return newStr;}




Using Assertions

• Asserting internal invariants:

• Asserting control invariants:

25

if (i % 3 == 0) { ... } else if (i % 3 == 1) { ... } else { assert i % 3 == 2 : i; ... }

void foo() { for (...) { if (...) return; // should occur at some point } assert false : "Execution should never reach this point!"}

Must only be satisfied within the

else block.

Always fails if statement is

reached




Using Assertions : Class Invariants / Effects

• A class invariant is a type of assertion that applies to every instance of a class at all times– Except when the instance is in transition from one consistent state

to another.

– Relationships among multiple attributes

– Should be true before and after any method completes

– Each public method and constructor should contain an assertion prior to its return (at each “exit point”)

26




Using Assertions : Class Invariants / Effects

• In Java, we use JavaDoc to describe effects:

27

/** * Increases the counter field by the values of the step parameter. * <p>Effect: the counter field is changed.</p> * @param step The value to increment the counter * @return Current value of the counter field. */public int inc(int step) { counter += step; return counter;}




Complex Assertions• We may use inner classes for complex assertions:

28

void foo(final int[] array) { // Inner class that saves state and performs final consistency check class DataCopy { private int[] arrayCopy; DataCopy() { arrayCopy = (int[]) array.clone(); }

boolean isConsistent() { return Arrays.equals(array, arrayCopy); } }

DataCopy copy = null;

// Always succeeds; has side effect of saving a copy of array assert ((copy = new DataCopy()) != null); ... // Manipulate array

// Ensure array has the same ints in the same order as before manipulation. assert copy.isConsistent(); }

Only copies when assertions are

activated




Recapitulate: Design by Contract

• Opportunities:– Simplification: We define correctness criteria only for a

certain part of the program.– Expectation: As long as all parts of the program fulfill their

contracts, the program will work (almost) correctly– Hope: If a failure occurs, we can find it by detecting

violations of some contract.

• Limitations:– Even though every contract is satisfied, the program might

not work correctly.• A contract might be semantically wrong• We might have forgotten to specify a certain condition in

the contract.

29

We need other methodologies to capture those cases!




Outline

30







Test Method

31

Systematic approach to testing building on a set of rules• In general: A systematic method to select

and/or generate tests:• Structural test methods• Functional test methods

Testing is only applicable with proper tool support




Basic Terminology• Test data:

– Input value that feeds an input parameter or an input variable with data.

• Test case:– A set of test data that leads to the complete execution of

the program under test– The expected results

• Test suite:– A non-empty set of test cases selected by a particular

criterion

• Test stub:– If the module under test is an operation of a class, it

cannot be tested in isolation.– Tester must program a test frame for the class that allows

an interactive call of the operation.

32




Basic terminology: Instrumentation

33

If a test case fails, we have to retrace which instructions of the test object have been executed in the test case locating errors

Tool-supported logging which parts of the code are executed

After execution, the counters/flags are evaluated

int maximum(int a, int b) { if (a > b) { thenBranch = true; return a; } else { elseBranch = true; return b; }}

inserted "counters"

inserted "counters"




Basic Terminology: Test Coverage

34

• Test coverage: ratio between actually executed test cases and theoretically existing test cases executed test cases existing test cases

• A coverage of 100% implies an "exhaustive test“- exhaustive testing is typically neither practical nor

useful

• Test end criterion– Sets the minimum goal for the test cases

TC =

Criterion for the selection of test cases




Test Methods• Structural Test Methods: Based on the code structure

(White Box Tests)– Test cases are only derived from the code structure– Completeness is judged by measuring the coverage

of structural elements (assignments, branches, etc.)– The specification of the code is ignored for judging

completeness

• Functional tests: Based on requirements (Black Box tests)– Ideally, the requirements are expressed as a formal

specification– Test cases are derived from the specification– Completeness is judged by referring to the

specification– The code structure is not referred to 35




Test Methods Overview

36

Black BoxBlack Box

Gray BoxGray Box

White BoxWhite Boxstructuraltesting

functionaltesting

boundary valueanalysis

Classification according to required information

control floworiented test

equivalenceclasses

checkuse cases

randomtest

datafloworiented test

In theory, Black Box methods. In practice,

often also uses internal structure




Control Flow Graph

• Control Flow Graph– represents control flow with a directed graph– Each node represents a statement– Directed edges connect nodes– The number of edges leaving a node is the node’s “fan

out”– Edge sequences describe potential control flow from start

node "i" to end node "j"• Branches

– directed edges • Path

– Sequence of nodes and edges, beginning with a start node and finishing with an end node

37




Control Flow Graph

38

• Control flow graphs always contain a start and an end node

• The end node has a fan out of 0

• All other normal nodes are on a path from start to end node

• Nodes with a fan out of 1 are statement nodes

• All other nodes, except the end node, are called predicate nodes




Control Flow Graphs (Example)

39

c = System.in.read(); // n1

while (c >= 'A' && c <= 'Z' && totalNumber<5) { // n2 totalNumber = totalNumber + 1; // n3 if (c=='A'||c=='E'||c=='I'||c=='O'||c=='U') { // n4 vowelNumber = vowelNumber + 1; // n5 } c = System.in.read(); // n6}

nend

n6

nstart

n1n1

n2n2

n3

n4

n5




Coverage tests

• Structural elements such as statements, branches, or predicates are used to define test targets.– Example: The branch coverage chooses

test data, such that all statements in the program are executed at least once.

• Implementing coverage tests– Instrument source code– The trace statements write a log– Execution of the log to generate a

coverage-report

40




Coverage Testing

41

Statement CoverageStatement Coverage

Path CoveragePath Coverage

Branch CoverageBranch Coverage Condition CoverageCondition Coverage

Multiple ConditionCoverage

Multiple ConditionCoverage

A B A implies B

Overview




Statement Coverage (C1)

42

n1

n2

n3

n4

n5

n6

n1

n2

n3

n4

n6

nstart

nend

• Goal: execute all statements

• Test case example– Input: <'A', '#'>

• Statement coverage helps to detect dead code

• Necessary but not sufficient

• Has very little practical relevance

n5




Branch Coverage (C1)

43

• Goal: Execution of all branches, i.e., all edges of the control flow graph.• Also called decision coverage: Every decision has

to have the values true and false at least once.

• Metric:

• Helps to detect non-executable branches• Fully contains statement coverage

• Involves neither a combination of branches nor complex conditions

• Regarded as the minimal test criterion

Cbranch =Nr. of branches executed

Nr. of branches in system




Branch Coverage (C1): Example

44

n1

n2

n3

n4

n6

nstart

nend

n5

n

n

n

n

n1

n2

n3

n4

n6

nstart

nend

n5

<'B','#'>

<'A','#'>




Condition Coverage (C2)

45

n1

n2

n3

n4

n5

n6

n1

n2

n3

n4

n6

nstart

nend

• test cases• <'A', 'E', 'I', 'O', 'U', '@'> <'€'>

• Contains no statement or condition coverage• Since both are minimal test criteria, a simple

condition coverage is insufficient

n5

all atomic conditions are true & false at

least once

else branch is never taken

if (c=='A' || c=='E' || c=='I' || c=='O' || c=='U')




Multiple Condition Coverage

46

n1

n2

n3

n4

n5

n6

n1

n2

n3

n4

n6

nstart

nend

• Test cases• <'A', 'E', 'I', 'O', 'U', '@'>

<'A', 'E', 'I', 'O', 'U', '€'><'A', 'E', 'I', 'O', 'U', 'A'><'A', '@'>, <'A', '€'>

• Contains branch coverage

• Combinatorial explosion: N atomic conditions 2N possibilities

• Certain combinations are impossible• Evaluation is perhaps aborted

(lazy evaluation)

n5

all combinations of true/false values for

atomic conditions required




Path Coverage (C7)

47

n1

n2

n3

n4

n5

n6

n1

n2

n3

n4

n6

nstart

nend

• Test cases• <'#'>, <'B', '#'>, <'A', '#'>

<'A', 'B', 'B', 'B', 'B'><'B', 'A', 'B', 'B', 'B'>, ...

• Contains branch coverage• Loops lead to potentially infinitely many test

cases• Highest chance of catching all errors• Entirely unfeasible in practice

n5

all combination of branches required




Coverage Testing: Summary

48

• Uncover deficits in the testing strategy• Identify dead code• Indicator of progress in testing

Advantages

• 100% coverage not practical• Code coverage does not imply full

coverage of functionality• Missing functionality is not recognized

Disadvantages

product is tested against itself




Functional Testing

• Motivation: insufficient to check a program only against itself – Derive test cases from the specification /

functionality description– Ignore internal code structure

49

Goal

• Check as much as possible of the functionality, avoiding redundancies

• Function coverage




Functional Testing

• Main Difficulties– Deriving the suitable test cases– Complete test of functionality is typically

not feasible

• Goal of Test Strategy – Choose test cases such that the probability

to find errors is high

• Maintaining Absence of Redundancy– Use of equivalence classes– Boundary value analysis

50




Test Methods Overview

51

Black BoxBlack Box

Gray BoxGray Box

White BoxWhite Boxstructuraltesting

functionaltesting

boundary valueanalysis

Classification according to required information

control floworiented test

equivalenceclasses

checkuse cases

randomtest

datafloworiented test




Equivalence Classes

• Motivation: reduce the number of test cases by partitioning test data into equivalence classes

• Assumption: for every representative piece of data from an equivalence class, the program will react in the same manner

• Examples– Classes: vowel consonant non-

letter– Repres.: E T %

– Classes: x<0 0x5 x>5– Repres.: -5 3 9

52




Boundary Value Analysis

• Motivation: errors are typically provoked by the extreme values within an equivalence class

• Assumption: the extreme representatives are not just as suitable, but especially useful to detect errors

• Examples

– Classes: x<0 0x5 x>5

– Repres.: -1 0 and 5 6

53




Test Methods: Summary

Advantages and Disadvantages:

• Structural Testing– finds: Abort errors, unreachable branches, infinite

loops, inconsistent conditions product checked against itself

• Functional Testing– finds: wrong answers

product checked against specification/requirements

54




Combined Strategy• Disadvantages of structural approaches:

– no discovery of unimplemented functionality– potentially many irrelevant test cases (with respect

to testing functionality) in order to just achieve code coverage

• ... and the disadvantages of functional approaches...– Based on the specification, which has a higher

abstraction level than the implementation– Achilles heels of implementation not used to

specifically target test cases at these– Typically, only a 70% branch coverage is achieved

• These disadvantages are complementary• One approach's advantages can be used to cancel out

the disadvantages of the other use a combined approach

55




Combined Strategy• Start with Functional Testing...

– Use specification to find equivalence classes– Determine boundary values– Create test cases Functionality systematically checked

• ...and follow up with Structural Testing– Analyze coverage achieved with the above– Identify neglected conditions / paths– Obtain desired coverage– (Relative) robustness of product ensured &

extra validation of the specification (to some extent)

56




Testing classes and subclasses

• The smallest independent testable unit of object-oriented systems is the class

• Operations of a given class are much too interdependent to test them in isolation.

• The test depends on the kind of class:– Normal class– Abstract class

57




Testing normal classes

1. Create an instrumented object of the class under test2. Check one operation after the other. First check

operations that do not change state and then the state changing operations.a. Through creation of equivalence classes and

border value analysis, test values are created from parameters• The object has to be brought into a state that is valid

for this test case– Through preceding test cases or by initializing it

for each test

b. After the operation is executed, the state must be checked and the computed results must be compared with the expected results.

58




Testing normal classes

3. Test each sequence of valid operations in the same class

– Ensure that you test each kind of object– All potential uses of an operation should be tested

under all relevant conditions

4. Check the coverage by using the instrumentation

– Uncovered areas should be covered by additional test cases.

59

telecooperation/rbg technische universität darmstadt copyrighted material; for tud student use only...

Documents

test coverage

test cases existing

test object

functional test methods

test cases tc

selection of test cases

t19 test methods overview

computer science i topic