october 26, 2010coms w41561 coms w4156: advanced software engineering prof. gail kaiser...

October 26, 2010 COMS W4156 1

COMS W4156: Advanced Software Engineering

Prof. Gail Kaiser

[email protected]

http://bank.cs.columbia.edu/classes/cs4156/


Topics covered in this lecture

• Integration Testing

• Code Inspection

• Some other testing issues


Integration Testing


What to Test?

• Unit test – test of small code unit: file, class, individual method or subroutine

Integration test – test of several units combined to form a (sub)system, preferably adding one unit at a time

• System (alpha) test – test of a system release by “independent” system testers

• Acceptance (beta) test – test of a release by end-users or their representatives


Overview of Integration Testing

• Performed after all units to be integrated have passed all black box unit tests

• Reuse unit test cases that cross unit boundaries (that previously required scaffolding standing in for another unit)

• White box testing might be combined with integration as well as unit testing (tracking coverage)


Test Harness (Scaffolding)

• test driver - supporting code and data used to provide an environment for invoking part of a system in isolation

• stub - dummy procedure, module or unit that stands in for another portion of a system, intended to be invoked by that isolated part of the system – May consist of nothing more than a function header

with no body– If a stub needs to return values, it may read and

return test data from a file, return hard-coded values, or obtain data from a user (the tester) and return it


Integration Strategies

• “Big Bang” vs. Incremental

• Top-down vs. Bottom-up vs. Sandwich


“Big Bang” Approach

1. Link everything together

2. Run Difficult to track down errors because bug

could be anywhere


A

DCB

FE G


TestA

TestA,B,C,D,E,F,G

TestB

TestD

TestE

TestC

TestF

TestG

A

DCB

FE G


TestA,B,C,D,E,F,G

A

DCB

FE G


Incremental Approaches

• Emphasis on testing interfaces • Focus on control/data flow paths where units call

other units or share data with other units• Put a few units together at a time• Ideally add only one unit at a time to a validated

base• Limits scope of errors to interactions between the

already-tested base and the unit(s) added


Top-down integration

1. Start with top-level module 2. Move downwards through control flow

hierarchy 3. Whenever add new modules, perform

regression tests on old modules to make sure no new errors introduced


Top-down integration• Depth-first integrates all modules in a major

vertical branch (wrt caller-callee module breakdown structure)

• Breadth-first integrates all modules directly subordinate at each level (or entire layer)

• Intuition in either case is to test program in relatively natural execution mode

• Minimizes need for drivers (but still need stubs, can often reuse from unit testing)


TestA

TestA,B,C,D

TestA,B,C,D,E,F,G

TestB

TestC

TestD

TestE

TestC

TestD

TestF

TestG

A

DCB

FE G


Bottom-up integration

1. Start with leaf modules (so need for stubs minimized)

2. Combine into larger subsystems– Move up caller-callee hierarchy to full logical

groupings– And, eventually, across logical grouping

boundaries


Bottom-up integration

• Intuition is to construct program from separately tested pieces

• Drivers needed to control and handle test case input/output (sometimes can reuse from unit testing)

• Tests utility modules early but tests control aspects of program relatively late


TestE

TestF

TestG

TestB,E,F

TestC

TestD,G

TestA,B,C,D,E,F,G

A

DCB

FE G


Sandwich integration

• Hybrid

• Top-down for highest levels of control hierarchy (within logical groupings and across system topology)

• Bottom-up for lower levels

• Reduces need for drivers and stubs


TestA

TestB,E,F

TestA,B,C,D,E,F,G

TestE

TestD,G

TestF

TestG

A

DCB

FE G

TestA,B,C,D


Summary of Integration Testing

• Performed after all units to be integrated have passed all black box unit tests

• Reuse unit test cases that cross unit boundaries (that previously required scaffolding standing in for another unit)

• White box testing might be combined with integration as well as unit testing (tracking coverage)


Code Inspection


Overview of Code Inspection

• Also known as walkthrough, review, etc.• An approach to verification and validation that does

not actually execute the code• Formal process for reading through the product as a

group and identifying defects• Potentially applied to all project documents, not

limited to source code• Used to increase software quality and improve

productivity and manageability of the development process


Inspection Roles

• The minimum roles required for an inspection are moderator/recorder, author/reader, and one other inspector, 3 participants, but 5 are “recommended”

• Moderator - Ensures that the inspection procedures are followed, and that the other inspectors perform their responsibilities for each step of the inspection process

• Recorder - Records and classifies all the defects detected at the inspection meeting, and assists the moderator in preparing inspection reports


Inspection Roles

• Reader - Leads the team through the product in a comprehensive and logical manner

• Author - Addresses specific questions that the reader is not able to answer (although the Author may also be the Reader), and detects defects based upon their special understanding of the product

• Inspector - All of the inspection team members are inspectors, regardless of their other roles


Inspection Process

• Define what will be inspected, • when it will be inspected, • who will do the inspection, • what data will be collected, and • what follow-up actions must be taken.


Actions Responsibility

Planning Author/Project Coordinator/Moderator

Preparation Moderator/Author

Inspection Moderator

Follow-up Moderator/Author


Planning Phase Prerequisites

• Minimum criteria, when applicable, includes:– Spelling and grammar checked (for documents)– Code should compile clean– Compliant with coding standards– Each inspection item is line numbered for easy

reference by inspectors


Planning Actions Responsibility

Assemble items to be inspected

Author

Select moderator, possibly from QA group

Project Coordinator (PC)

Review inspection items for completeness

Moderator

Select inspection team, assign roles, notify

PC

Schedule inspection Moderator


Preparation Actions Responsibility

Review all items to be inspected Inspectors

Record identified defects on individual Defect Log

Inspectors

Send individual Defect Log to Moderator 48 hours prior to inspection meeting

Inspectors

Combine individual Defect Logs in a logical order

Moderator

Disperse the combined Defect Log to all participants for review 24 hours prior to the meeting

Moderator


Preparation Phase

• If more than a few Defects are found individually, then inspection may be postponed

• A redlined copy of the inspection item is returned to the author after the inspection to aid in defect location


Meeting Actions Responsibility

Review rules for inspections Moderator

Confirm inspection preparation Moderator

Systematically review (walkthrough) the product being inspected

Reader

Present and clarify defects previously and newly identified

Author, Inspectors

Categorize valid defects, produce revised Defect Log

Recorder

Determine results of the inspection (Accept, Conditional, Re-inspect)

Moderator

Determine estimated rework effort and completion date

Author

Inform PC of inspection disposition Moderator


Rules for Inspections

• Maximum time scheduled for an inspection meeting should be about 2 hours (according to Michael Fagan, who studied the concept thoroughly at IBM - http://www.mfagan.com/resource_frame.html)

• Start meeting on time and move it along so that it finishes on schedule

• Defer arguments over style or technique• Solutions or alternatives may be mentioned, but avoid

spending too much time on group rework• Address the product directly, do not attack the author


Inspection Disposition

• Accepted - The product is complete, without any further verification of rework. Only a very few trivial defects that can be left to the Author to correct.

• Conditional - Conditionally accept the product, with verification of the rework by the Moderator. In this case there are a few major defects, and rework is not expected to create any substantial changes in the product

• Re-inspect - Re-inspect the Author’s rework. There are either a substantial number of major defects, or rework that will change the original product. This disposition requires that the rework be examined by the Moderator, the Author, and at least one other member of the inspection team in a reiteration of the inspection meeting.


Follow-up Actions Responsibility

Correct or resolve all noted defects Author

Validate corrections made by author (conditional disposition)

Moderator

Complete Inspection Report Moderator/Recorder

Post Inspection Report to project repository

Moderator

Record metrics (e.g., Size of materials inspected; Defects by severity, class, and type; Preparation times prior to inspection; Estimates of rework effort)

Moderator


Code Inspection Resources

NASA: http://software.gsfc.nasa.gov/AssetsApproved/PA2.4.2.1.doc

SEI’s Personal Software Process: http://www.ipd.uni-karlsruhe.de/mitarbeiter/muellerm/PSP/Dokumente/DefTyp/DefTyp.html

IBM: http://www.research.ibm.com/softeng/ODC/DETODC.HTM


Other Testing Issues


Testing Database

• What program unit(s) was(were) tested • Actual inputs • Expected outcomes• Actual outcomes• Include global variables, files, other state• Who performed tests • Who was informed of errors


Links to Other Materials

• Configuration information (e.g., module versions, dependencies on other systems and their versions, platform details)

• Test scripts or programs• Scaffolding used• Bug report• Error analysis• Change request• Anything else...


Why Keep Records?

• Some parts are necessary for regression testing

• Same bug may be reported multiple times by different customers

• Track staff responsibility - who did what

• Process measurement and improvement - relate quality/quantity of tests to quality of program


Regression Testing After Changes

• Change is successful, and no new errors are introduced

• Change does not work as intended, and no new errors are introduced

• Change is successful, but at least one new error is introduced

• Change does not work, and at least one new error is introduced


Regression Testing After Every Change

• If rerun entire test suite after every small change

• And now one or more tests fail

• You know exactly who broke the code – you did!

• And exactly what broke the code – that last change


Economics of Testing

Costs associated with testing software

• Planning testing process

• Writing test cases

• Setting up equipment

• Systematically executing test cases

• Follow-up on problems identified

• Removing [most] defects


Economics of TestingCosts associated with shipping software defects

to customers • Customers lose faith in developers and get

very angry• Customers can lose a great deal of money if

their system goes down• Expensive to obtain specific information about

customer problems and to find and fix their bugs, possibly on-site


Economics of Testing

Also need to consider the loss potential of each individual project developer’s work

• Quality is much more important for safety- or mission-critical software than for consumer software (e.g., video games)

• Balance the cost of testing versus the cost of software failures


Economics of TestingTo minimize costs, one goal of testing must be to

uncover as many defects as possible with as little testing as possible

• It is simply impossible to exhaustively test every possible input-output combination of the system

• There are simply too many permutations and combinations

• Define test cases that have a high likelihood of uncovering defects

Upcoming Assignments


Upcoming Assignments

• First Iteration Demo Wednesday 3 November – Thursday 11 November.

• First Iteration Final Report due Friday 12 November.

• Midterm Exam posted Thursday 11 November and due Friday 19 November.


COMS W4156: Advanced Software Engineering

Prof. Gail [email protected]

http://bank.cs.columbia.edu/classes/cs4156/


october 26, 2010coms w41561 coms w4156: advanced software engineering prof. gail kaiser...

Documents

unit testingcoms w4156october

test data

itcoms w4156october

sandwichcoms w4156october

units addedcoms w4156october

timesystem alpha test

cross unit boundaries

system release