testing - william & mary

Testing

1

Software is Buggy!• On average, 1-5 errors per 1KLOC• Example: Windows 2000– 35M LOC– 63,000 known bugs at the time of release– 2 bugs per 1000 lines

• Infeasible: – 100% correct software for mass market.

• But: –We must verify software as much as

possible2

3

SW Faults, Errors & Failures

• Software Fault : A static defect in the software (i.e., defect, bug)

• Software Error : An incorrect internal state that is the manifestation of some fault

• Software Failure : External, incorrect behavior with respect to the requirements or other description of the expected behavior

A Different View: Error, Faults, Failures

Fault & Failure ModelThree conditions necessary for a failure to be

observed1. Execution/Reachability: The location or locations

in the program that contain the fault must be reached

2. Infection: The state of the program must be incorrect

3. Propagation: The infected state must propagate to cause some output of the program to be incorrect

PIE model

Execution Based Testing• No proof by example:

“Program testing can be a very effective way to show the presence of bugs but is hopelessly inadequate for showing their absence” [Dijkstra]

• Testing:–Only validation technique for non-

functional requirements– Should be used in conjunction with static

verification to provide full V&V coverage

Today, QA is mostly testing• QA: Quality Assurance

• “50% of my company employees are testers, and the rest spends 50% of their time testing!”

Bill Gates 1995

7

Typical Software License• 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO

WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

• 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

8

What is testing?• Testing ==

To execute a program with a sample of the input data• Dynamic technique: – program must be executed

• Optimistic Approximation: – Program exercised with a (very small)

subset of all the possible input data – Assumption:

behavior with other input is consistent with the behavior shown for the selected subset of input data 9

Role of Testing• Common practice in engineering – Design a drug – Manufacture an airplane ...

• Why? – Because our ability to predict how our

creations will behave is imperfect – We need to check our work, because we

will make mistakes

10

The Purpose of Testing• Two purposes:1. Find bugs– Find important bugs

2. Elucidate the specification– When testing the

prototype or strawman

• Testing requires a certain mindset– Want to break the code

• Good testing is hard work– Requires real insight into

the nature of the system– Will help elucidate the

spec

11

Testing Real Software is Hard (E.g. Word)

• Many input forms – mouse, keyboard, tablet

• Many document formats – .doc, .html

• Outputs – printer, screen

• External libraries – Fonts, Dictionaries, Languages

• Interoperation – VB, Excel, email

• Many features – 100+ commands – 34 toolbars

• Constraints – Huge user base

12

Independent Testing• Programmers have a hard time

believing they made a mistake – Plus a vested interest in not finding

mistakes• Design and programming are

constructive tasks – Testers must seek to break the

software

13

Independent Testing• Wrong conclusions: – The developer should not be testing at all

• Recall “test before you code”

– Toss the software over the wall for testing• Testers and developers collaborate in developing the test

suite

– Testers get only involved once software is done – Testing team is responsible for assuring quality

• Quality is assured by a good software process

14

Methods of Testing• Test to specification:– Black box– Data driven– Functional testing– Code is ignored: only use specification document

to develop test cases• Test to code:– Glass box/White box– Logic driven testing– Ignore specification and only examine the code

Types of Testing• Unit Testing

– Done by programmer(s)– Generally all white box– Automation desirable for repeatability

• Integration Testing– Done by programmer as they integrate their code into code base– Generally white box, maybe some black box– Automation desirable for repeatability

• Functional/System Testing– It is recommended that this be done by external test group– Mostly black box so that testing is not ‘corrupted’ by too much

knowledge– Test automation desirable

• Acceptance Testing– Generally done by customer/customer representative in their

environment . . . Definitely black box

Unit Testing• Focus on smallest unit of design– A procedure, a class, a component

• Test what?– Local data structures– Basic algorithm– Boundary conditions– Error handling

• May need drivers and stubs• Good idea to plan unit tests ahead

19

Integration Testing• If all parts work, how come the whole doesn’t?• For software, the whole is more than the sum of the

parts– Individual imprecision is magnified (e.g., races)– Unclear interface design

• Don’t try the “big bang” integration!• Do incremental integration

– To simplify error localization, systems should be incrementally integrated

– Top-down integration– Bottom-up integration

20

Top-Down Integration• Test the main control module first• Slowly replace stubs with real code– Can go depth-first• Along a favorite path, to create quickly a

working system (i.e., happy path)– Or, breadth first

• Problem: you may need complex stubs to test higher-levels

21

Bottom-Up Integration• Integrate already tested modules• No stubs, but need drivers– Often the drivers are easier to write

• Example: – Financial code that depends on subroutine

for computing roots of polynomials–We cannot test the code without the

subroutine• A simple stub might not be enough

–We can develop and test the subroutine first

• Plan for testability ! 22

Validation Testing• Culmination of integration testing

– The software works, but does it do what we need?• Run acceptance tests

– Get your customer to define them• Alpha-testing (in controlled environment)

– With developer looking over the shoulder• Beta-testing

– At end-user sites

23

Regression Testing• Idea–When you find a bug,–Write a test that exhibits the bug,– And always run that test when the code

changes,– So that the bug doesn’t reappear

• Without regression testing, it is surprising how often old bugs reoccur

24

Regression Testing (Cont.)• Regression testing ensures forward progress– We never go back to old bugs

• Regression testing can be manual or automatic– Ideally, run regressions after every change /

check-in– To detect problems as quickly as possible

• But, regression testing is expensive– Limits how often it can be run in practice– Reducing cost is a long-standing research

problem25

Regression Testing (Cont.)• Other tests (besides bug tests) can be

checked for regression– Requirements/acceptance tests– Performance tests

• Ideally, entire suite of tests is rerun on a regular basis to assure old tests still work

26

Other Forms of High-Level Testing

• Security testing– Red-team testing

• Performance testing– E.g., real-time systems

• Stress testing …

27

Stress Testing• Push system into extreme situations

– And see if it still works . . .

• Stress– Performance

• Feed data at very high rates– Interfaces

• Replace APIs with badly behaved stubs– Internal structures

• Works for any size array? Try sizes 0 and 1.– Resources

• Set memory artificially low.• Same for # of file descriptors, network connections, etc.

28

Stress Testing (Cont.)• Stress testing will find many obscure bugs– Explores the corner cases of the design“Bugs lurk in corners, and congregate at

boundaries”

• Some may not be worth fixing– Bugs too unlikely to arise in practice

• A corner case now is tomorrow’s common case– Data rates, data sizes always increasing– Your software will be stressed 29

A Problem• In an ideal world, we should have formal

specification, but in most cases– Too complex– Must change with the code– May be incorrect too!

• In an ideal world, we should also have perfect verification, but in most cases– Too expensive– Too imprecise– Many interesting properties are undecidable– We must accept inaccuracy!

What if we exhaustively test our program?

30

Exhaustive Testing?• How long would it take

(approximately) to test exhaustively the following program?

int sum(int a, int b) { return a + b; }

232 X 232 = 264 ~ 1019 tests

Assume 1 test per nanosecond (109 tests/second)

We get 1010 seconds…About 600 years 31

What kind of testing?• Manual– Good for exploratory tests– Good for testing GUI • automating GUI testing is important research

area–Manual regression testing is BORING

• Automatic– Test is a program– Test is created by a tool that records user

actions 32

Test automation• Tests are code or scripts (which is code)• Often more test code than production

code• Test code is boring– build some complex data values– run a function– check the result

33

Black-Box Testing• An approach to testing where the program

is considered as a ‘black-box’• The program test cases are based on the

system specification • Test planning can begin early in the

software process

Black-Box Testing

Important Consideration for Black Box Test Planning

• Look at requirements/problem statement to generate test cases

• Test cases need to be traceable to a requirement• You must write the repeatable test case so

anyone on the team can run the exact test case and get the exact same result/sequence of events– The inputs must be very specific

• Example: “Students who receive a grade of 70 or higher pass the exam”

• Correct test cases: Grade = 80; Grade =20• Incorrect test cases: “input a passing grade” “input a

failing grade”– The expected results must be very specific. “Pass” “Fail”

Equivalence Class Partitioning• Divide your input conditions into groups (classes)

– Input in the same class should behave similarly in the program

• Be sure to test a mid-range value from each class• Example from Monopoly game: for tests of “Go to

Jail” the most important thing is whether the player has enough money to pay the $50 fine– Test input values clearly in the two partitions: 25 and

75

Boundary Value Analysis• Focus on boundaries . . . because a greater

number of faults tend to occur at the boundaries of the input domain– Range input, a to b, test with a, b, a-1, a+1, b-1,

b+1 if integer range; otherwise, slightly less than a and slightly more than b.

– If you can only have a certain quantity (q) of something, try to create q-1, q, q+1

Dirty/Failure Test Cases• Can something cause division by zero?• What if the input type is wrong (You’re

expecting an integer, they input a float. You’re expecting a character, you get an integer)?• What if the customer takes an illogical

path through your functionality?• What if mandatory fields are not entered?• What if the program is aborted abruptly or

input or output devices are unplugged?

White-Box Testing

White-Box Testing• Sometime called structural

testing or glass-box testing• Derivation of test cases

according to program structure• Knowledge of the program is

used to identify additional test cases

• Objective is to exercise all program statements (not all path combinations)

Black-box Testing Example Search Routine Specification

procedure Search (Key : ELEM ; T: ELEM_ARRAY; Found : in out BOOLEAN; L: in out ELEM_INDEX) ;

Pre-condition -- the array has at least one element T’FIRST <= T’LAST Post-condition -- the element is found and is referenced by L ( Found and T (L) = Key) or -- the element is not in the array ( not Found and not (exists i, T’FIRST >= i <= T’LAST, T (i) = Key ))

Search Routine - Input Partitions


• Inputs which conform to the pre-conditions

• Inputs where a pre-condition does not hold

• Inputs where the key element is a member of the array

• Inputs where the key element is not a member of the array

Question• Which test cases based on input

partitions should we derive?

Testing Guidelines – Sequences

• Test software with sequences which have only a single value

• Use sequences of different sizes in different tests

• Derive tests so that the first, middle and last elements of the sequence are accessed

• Test with sequences of zero length

Sorting Example• Example: sort (lst, n)– Sort a list of numbers– The list is between 2 and 1000 elements

• Domains: – The list has some item type (of little concern)– n is an integer value (sub-range)

• Equivalence classes;– n < 2– n > 1000– 2 <= n <= 1000

Sorting Example• What do you test?• Not all cases of integers• Not all cases of positive integers• Not all cases between 1 and 1001

• Highest payoff for detecting faults is to test around the boundaries of equivalence classes

• Test n=1, n=2, n=1000, n=1001, and say n= 10

• Five tests versus 1000

White Box Testing - Binary Search int search ( int key, int [] elemArray){ int bottom = 0; int top = elemArray.length - 1; int mid; int result = -1; while ( bottom <= top ) { mid = (top + bottom) / 2; if (elemArray [mid] == key) { result = mid; return result; } // if part else { if (elemArray [mid] < key) bottom = mid + 1; else top = mid - 1; } } //while loop return result;} // search

Binary Search Equivalence Partitions

• Pre-conditions satisfied, key element in array• Pre-conditions satisfied, key element not in

array• Pre-conditions unsatisfied, key element in array• Pre-conditions unsatisfied, key element not in

array• Input array has a single value• Input array has an even number of values• Input array has an odd number of values

Binary Search Equivalence Partitions

Binary Search - Test Cases

Three Important Testing Questions

• How shall we generate/select test cases?

• Did this test execution succeed or fail?

• How do we know when we’ve tested enough?

65

testing - william & mary

Documents