software engineering lecture 11 vladimir safonov, professor, head of laboratory st. petersburg...

SoftwareEngineering

Lecture 11

Vladimir Safonov,Professor, head of laboratory

St. Petersburg University

Email: [email protected]

WWW: http://user.rol.ru/~vsafonov

(C) Vladimir O. Safonov, 2004 2

Software testing

• Testing is systematic checking the program for the purpose of detecting and fixing bugs, and also for the purpose of investigating its functional characteristics and the resources taken

• E. Dijkstra: “Testing can prove the presence of bugs only but would never prove their absence”

• The general task of formal verification of programs (instead of their testing) is not yet solved.

Successful are only verification efforts for certain classes of programs (embedded systems, telecommunication systems)

• “Testing vs. debugging” : testing and debugging is not the same


Kinds and methods of testing

• Testing all bug fixes (regression testing)

• Compatibility testing (testing of conformance to the specification); examples – JCK for Java; BSI 5.5 for Sun Pascal Compiler

• Performance testing, or benchmarking – SPEC, etc.; testing of time / memory resources taken by the program, and the performance it demonstrates

• Stress testing – testing for stability under the circumstances of very extensive use: millions of object generations; millions of requests to the server, etc.


Testing strategies

• White box – the source of the program under testing is available. The goal is to achieve maximum test coverage (block coverage, branch coverage, condition coverage, method coverage, etc.)

• Black box – the source of the program under testing is unavailable. The goal is to achieve 100% method coverage, and to test against “reasonably maximum” of combinations of the argument values


Strategies to choose the sets of arguments for testing (G. Myers)

• Boundary values – testing for boundary values, like 0, 1, -1, null

• Equivalence class partitioning – choose one value from each typical equivalence class (a subset of argument values), such that for each value from that equivalence class the module being tested is supposed to behave the same way.

Example: GCD(x, y), where x > 0 is some concrete integer value

• For testing P(x, y) all in all M * N sets of argument values are possible, where M is the number of selected values of X; N is the number of selected values for Y


Classification of the kinds of testing

• Static and dynamic testing• Static testing (in average helps to find 70% of bugs) :

rewriting the code; individual through code review, group through code review,

group inspection – analysis against typical bugs; analysis using source code verifier (lint, Flexelint, etc.)

• Typical bugs: - using undefined (“garbage”)values of variables - array index out of bounds - pointer bugs (nil/null, etc.) - interface bugs: calling P(y, x) instead of P(x, y), where

x, y are of the same type


Organization of testing at commercial companies

• Nightly, weekly, monthly building / testing (“heartbeat tests”)• Test engineers: SQE (Software Quality Engineers) – developers of tests and testing

tools; SQA engineers (software quality assurance) – as a rule, just “testers” (those who “presses buttons”, runs the ready test suites, tests GUI by hand, etc.)

• Test plan – the plan of testing prepared by the SQE manager• Testing tools: - test harness– a tool for running tests and displaying and analyzing the results:

JavaTest, JUnit, etc. - test base – set of test suites consisting of tests and their configuration and other

accompanying files: * test – set of test cases * golden file(s) – files with expected results (BUT: when random testing is necessary,

instead of golden files “check correct” functions are used * configuration files, exclude lists, etc. - test workspace(s) – the workspace for storing the log and the results of testing –

KEEP IT APART FROM THE TEST BASE• Typical approach (bug) of small companies – “no resources for SQE” (causes poor quality of the product)


Maintenance (sustaining)

• The most resource consuming stage of software lifecycle

• Includes: installing new versions of the product, training users and answering their questions, fixing bugs, product enhancement based on users’ requests

• As a rule, the resources allocated for maintenance are too small (“0.25 engineer”)

• Please note that the product dies if it’s not maintained

• Maintenance is very often performed by the engineers different from the authors of the original version of the product

• Typical technological issue: locate an aspect implementing the functionality to be fixed


Bugs: evaluation and fixing

• Bug tracking database – the database containing information on all known bug reports. It is proprietary and confidential; available for open / shared source products only (e.g. bugzilla)

• Bug id – the ordering number of the bug

• Synopsis – brief description of the bug, with the appropriate keywords

• Description – detailed description of the problem

• Priority – the urgency of the bug; defined by the customer and decreased by product manager before shipping the release

As a rule, priority 1 means the bug is extremely urgent; the bug shoul be fixed not later than in a week

• Severity – the internal functional seriousness of the bug


Bugs: evaluation and fixing (cont.)

• The stages of processing a bug: - submitted– the initial stage; performed by an CTE (Customer

Technical Escalations) engineer - accepted– initially processed by the responsible manager of the

project who appointed the responsible engineer for fixing the bug - evaluated– the responsible engineer investigated the problem and

entered into the bug tracking database the result of his evaluation, in particular, his suggested fix – the proposed correction of the bug (in the form of collection of updates of the sources)

- fixed– the bug fix is entered into the source code workspace and tested

- integrated– the fix is integrated into the master source code workspace; for P1 and P2 bugs, there should be made a patch (a new version of the product with the appropriate corrections)


Bugs: the alternative outcomes of their processing

• Fixed/integrated/verified – the bug fix is integrated and tested by a release engineer (independent testing)

• Closed because:

- fixed and verified

- the bug is a duplicate of any already known bug

- not reproducible

- will not fix (no resources to fix the bug); undesirable

No matter the bug is closed, the bug report is stored on the bug tracking database


Manufacturing the product

• Versions: dot-release (2.0, 2.1, …) – contains major new features; dot-dot-release (“bug fix release”) – contains new bug fixes only

• Early Access, Alpha. Beta, FCS (First Customer Shipment)– the stages of manufacturing the product; FCS => no P1 bugs; Beta => no P1/P2 bugs

• Alpha and Beta testing (by universities and individual volunteers)

• Before each release – QA cycle• Documentation + product notes (the latter document

issued just before the release and contails the list of known but not-yet-fixed bugs)


Software Process

• Chief Programmer’s Team – by F.P. Brooks (IBM, 1970s): surgeon, the second pilot, adnimistrator, language expert, tester, toolmaster, archive keeper, editor, secretary of administrator, secretary of editor – total 10 roles (persons if possible)

• Capability Maturity Model (CMM) – CMU SEI; Motorola (level 5): five levels of software process organization – initial, repeated, documented, supported by tools, optimizing

• Extreme Programming (XP) – paired coding, developing tests before implementing new functionality, refactoring, collective code ownership

• Microsoft’s approach (software project survival guide): tiger teams

• Psychological and moral issues


Email as a tool for organizing software process

• Email is a working tool; all project decisions should be discussed by email

• To: the concrete engineer; Cc: the manager and the technical group

• Subject: should be explicitly present; should be very concrete

• Friendly reminder (up to 10-15 times); please be patient

• Email should be used for: technical discussions, event notifications; bug reports notifications; source code workspace notifications (automated); forwarding the working documentation; shipping the updates in the product (automated; most important for remote work)

• Politeness: Dear John, <content> Thanks, Ivan

• No rude, no spam. Each email should be concrete and up to the place

software engineering lecture 11 vladimir safonov, professor, head of laboratory st. petersburg...

Documents