chapter 12: software testing & quality assurance instructor: paul k chen

Chapter 12: Software Testing & Quality Assurance

Instructor: Paul K Chen

TopicsTopics

I. An Overview

II. The Testing Process

III. Software Quality Assurance

VI. Quality Measurement

V. Concluding Remarks

I. An OverviewI. An Overview

I.1 Definitions

I.2. Related Terms and Definitions

I.3 Typical Software Quality Factors

I.4. Testing vs. Quality Assurance

I.5 The Three Levels of Testing

I.1 DefinitionsI.1 Definitions

Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.

Quality is not intangible

The purpose of testing is to make quality visible.

Testing is the measurement of software quality.

I.2. Related Terms and I.2. Related Terms and DefinitionsDefinitions

Alpha Testing Alias for functional testing performed by developers. Beta Testing Alias for functional testing performed by users. Regression Testing Testing is performed after functional improvements or

repair to the program. Its purpose is to verify that a modified system still meets requirements.

Black Box Testing The tester is completely unconcerned with the internal

behavior and structure of the program.

Related Terms and DefinitionsRelated Terms and Definitions

Functional Testing

Testing is performed to find discrepancies between the program and its external specifications.

White Box Testing

Testing is focused on the internal processing and structure of the program.

Quality Assurance

A planned and systematic pattern of all actions necessary to provide confidence the product conforms to established requirements.

I.3. I.3. Typical Software Quality Typical Software Quality FactorsFactors

Functionality (External factors)

Correctness; Reliability; Usability; Integrity Engineering (Internal quality)

Efficiency; Testability; Documentation; Structure Adaptability (added dimension)

Reusability; Expandability (flexibility);

Maintainability

I.4. Testing vs. Quality I.4. Testing vs. Quality AssuranceAssurance

Testing and quality assurance must work together for the continual improvement of the developmental and maintenance process.

The test personnel should record the defect by type which should be given to quality assurance at appropriate testing checkpoint

Testing vs. Quality AssuranceTesting vs. Quality Assurance

Quality assurance should use these defect lists to identify troublesome parts of the test, development, and maintenance process. The sharing of this information is to eradicate the cause of the most frequently occurring defects.

In addition, quality assurance must use this information only for improvement in the process, and should not present it to other individuals as this would be a breach of trust in the partnership relationship.

I.5. The Three Levels of Testing-I.5. The Three Levels of Testing-Unit TestingUnit Testing

Unit testing Testing of individual programs as they are written.

Objective: Confirm that module is coded correctly. Who does it: usually the programmer. What is tested: functions are exercised (black box);

code may be exercised (white box); Extremes and boundaries are explored.

When completed: Usually when programmer feel comfortable and has no known defects.

Records: Usually not recorded.

The Three Levels of Testing-System The Three Levels of Testing-System TestingTesting

System Testing Testing of groups of programs Objective: Assemble modules into a working system.

Determine readiness for acceptance testing. Who does it: Team leader or test group. What is tested: system requirements; internal and external

interfaces. When completed: Usually when majority of requirements

met; no major defects remain. Tools or Aids: Test cases; test data generator. Records: Defects discovered usually logged; test cases

maintained.

The Three Levels of Testing-The Three Levels of Testing-Acceptance TestingAcceptance Testing

Acceptance Testing Acceptance testing begins when system testing is complete.

Objective: Evaluate readiness for use. Who does it: End users or their agents. What is tested: Major functions, documentation and

procedures. When completed: Usually when user is comfortable or

when test set runs successfully. Tools or Aids: Comparators. Records: Spotty; some highly formalized.

II. The Test ProcessII. The Test Process

II.1. Define Testing Strategy

II.2 Develop Testing Plan

II.3 Execute Tests

II.4 Summarize and Report Testing Results

II.1. Define Testing StrategyII.1. Define Testing Strategy

Definition:

Testing strategy is the planning of test activities and the

subsequent systematic control of these activities. Both the

management and key staff must agree upon a purpose

and direction.

Testing StrategyTesting Strategy

Test strategies encompass at least:

Objective: What you try to achieve.

Scope: Define what areas you want to cover.

Approach: How testing is performed to achieve the goal.


Specifically test strategies include the following:

Testing Methods- Inspection; Analysis; Demonstration; Test

a. Inspection using visual examination. b. Analysis: Verification using comparison with

expected results. c. Demonstration: Verification by performing the

function the result of which is observable. d. Test: Verification by systematic control and

exercising.


Cost & Benefits

Cost: Equipment; personnel; education; schedule. Benefits: Reliability; maintainability; measurable

Limitations

Time; cost; system complexity

Organizations

Development; user; independent


Verification strategies by area Requirements tractability; design conformance to

standards, architecture, directions, and code.

Verification strategies by approach Walkthrough; top down; bottom up; thread; hybrid Walkthrough: An informal meeting held to verify the

validity of a product. Area to review: Functional and/or technical format; 3-7 people among peers using team dynamic.

ISO 9001ISO 9001

The ISO 9001 series of International Standards for Quality management and quality assurance has been adopted in more than 90 countries and is being implemented by thousands of manufacturing or service organizations in both public and private sectors. One of the most successful series of standards in the history of ISO (International Organization for Standardization), ISO 9000 has generated much publicity and made the name of ISO known to a wider business community than the specialists directly concerned with technical standards.

II.2. Develop Test PlansII.2. Develop Test Plans

What is it? Testing plans are the realization of testing strategies. It is an approved statement of work for all testing activities.Test plans can be written for various levels of testing:

High level Overall project testing plan System test plan Subsystem test plan Integration test plan Low level Unit test plan

Develop Test Plans-ContinuedDevelop Test Plans-Continued

Selection of test items Do a work breakdown structure by product and task. Test personnel and organization Responsibility matrix Test facilities and resources Hardware and software requirements Testing schedule Testing libraries Test data Test training Test tools

Task Breakdown StructureTask Breakdown Structure

System

Task Component

A

Task Component

B

Task A Task B Task C Task D Task E

Develop Test Plans-ContinuedDevelop Test Plans-Continued

Test (control) procedures

Test readiness review Certification process Tractability matrix

Measurements Validation methods Configuration management policies Testing problem reporting procedures Testing logs

II.3. Executing TestsII.3. Executing Tests

Design test case

Select test tools

Select test conditions and data

Set up test case folios

Test Case DesignTest Case Design

Black box approach: (requirements-based)Characteristics: Testing functionality of the system and ignoring the

internal structure of the system. Tests are derived from specifications.

White (glass) box approach: (design-based)Characteristics: Focusing on the structure of the system and ignoring its

specifications Tests are derived from program structure

Black Box ApproachBlack Box Approach

Advantages: Functions are forced to be known and enumerated.Disadvantages: Usually requires a lot of tests. Specifications may not be complete.Techniques Decision tree diagram State transition diagram (for real time system) Input/output space partitioning (probabilities/combination) Boundary value analysis Walkthrough

Black Box Approach-Black Box Approach-TechniquesTechniques

Scientific Cause/effect graph

Input/output space partitioning

Boundary analysis

Random testing

Non-scientific Error guessing

Start with most scientific approach; build a base of test

cases; Complement your base by adding additional cases

derived from the other techniques.

Black Box Techniques- Black Box Techniques- Cause/Effect GraphingCause/Effect Graphing

Cause/effect graphing (decision tree/decision table)

Method: Analyze the specifications to identify all explicit and

implicit causes and effects. Give each cause and effect an arbitrary number.

Draw the cause/effect graph (decision tree). Convert the graph (tree) to a decision table.

Decision TableDecision Table

It shows that transformation between a set of inputs and a

set of outputs. Decision tables are a useful mini-

specification technique when the control processing is

better expressed as a “look-up tables” than as an

algorithmic procedures.

1. $ amt S small (0-$1.999) . L large (at least $2,000)2. Per share cost C low cost (0 -$19/share) X expensive (at least $20/share3. Type of lot E even lot (multiple of 100 share)

Dollar AmountPer share costType Lot

1 2 3 4 5 6 7 8

S S S S L L L L

C C C CX X X XE E E EO O O O

5 % of base

3 % of base

10 % of base

5 % of base

1 % of base

yes yes yes yes

yes

yes

yes

yes

yes

yes

yes

yesyes

yes

yes

YesYes YesYes

Yes

Yes

Decision TreeDecision Tree

The decision tree is a variant of the decision table that

depicts a graphical view of the conditional selection

process used to determine a set of actions from a set of

inputs. The decision points are the nodes in the tree; the

actions are found at the leaves.

CommissionPolicy

SmallAmount

LargeAmount

LowCost

expCost

LowCost

expCost

Even Lot

Even Lot

Odd Lot

Odd Lot

Even Lot

Odd Lot

Even Lot

Odd Lot

5 %+10 % of Base

5 %+11 % of Base

5 %+6 % of Base

5 %+5 % of Base

5 %+10 % of Base

5 %+11 % of Base

5 %+5 % of Base

5 %+6 % of Base

Black Box Black Box Techniques-Input/Output Space Techniques-Input/Output Space PartitioningPartitioning

Methods: 1. Analyze the specifications to identify all input and output

conditions. Partition these conditions into classes. These classes are further divided into two types, valid vs. invalid.

Case in point:If input conditions have a range 1-99, specify one valid class (1-99) and two invalid classes (<1 and > 99)If input conditions have a set of values, (for ex., passing grades A, B,C), specify one valid equivalence class for each and one invalid equivalence class.

Black Box Black Box Techniques-Input/Output Space Techniques-Input/Output Space PartitioningPartitioning

2. Identify the equivalence classes for output conditions and partition into classes. These will also fall into two types-valid vs. invalid.

3. Identify “cases” that will be representative for each of the classes identified.

Black Box Techniques- Black Box Techniques- Boundary value analysisBoundary value analysis

Boundary value analysis

Boundary conditions are those values directly above or

below the edges of the input equivalence or output

equivalence classes. For example:

If a range 1-99 is specified, test for 0,1,99,100. If a number 1 is specified, test for 1,1, -1, 1, 1.001, .999,

etc.

Black Box Techniques- Black Box Techniques- Random Random TestingTesting

Random testing

Uses a randomly selected subject or all possible input.

Better yet, use statistical sampling techniques.

Error guessing

White Box ApproachWhite Box Approach

Advantages: Integration dependent errors can be found.

Disadvantages: Tests can not find the functionality of the program. Number of tests may be prohibitive.

White Box TechniquesWhite Box TechniquesTechniques Path testing Each executable path through a program is tested at

least once. Total path coverage is best but not enough. Branch testing each entry point is invoked at least once; Special values

testing. Boundaries in the input space are explored and potential exceptions or end-point conditions in the code are targeted at by the choice of input data.

Walkthrough A review process in which a designer leads one or more

others through a segment of design or code he or she has written.

Select Test ToolsSelect Test Tools

Program logic analyzer

Produces hierarchical charts depicting program logic structure (structure chart)

Analyzes source code for logical structure and produces reports flagging logic discrepancies.

Test Data Generator

Collects data from existing files.

Collects data based on file attributes and program. For example, Pro/Test.

Select Test ToolsSelect Test Tools

Captive/Replay Tools

Recreates online user transactions by capturing screen input, for ex., Traps.

File comparators

Compares contents of two files and identifies discrepancies.

Select Test DataSelect Test Data

Criteria: reliability and validity

Reliability: If any set of test data selected will reveal any incorrectness of the program.

Validity: If it is simply able to select a test data set that will demonstrate the incorrectness of the program.

Two orthogonal but complementary approaches to test data selection: Black box treatment vs. White box treatment.

Set Up Test Case FoliosSet Up Test Case Folios

The test case folio is simply a folder or a document

containing information on a case or related group of

cases. Possible contents:

1. Identification for the test group or test case.

2. Testing objectives.

3. Description of the situation being tested.

4. Test data (or a reference to where it is or how generated).

Set Up Test Case FoliosSet Up Test Case Folios

5. Expected or required results.

6. Entrance criteria (What has to happen before the test can start).

7. Setup and execution procedures.

8. Exit criteria.

9. Responsibility.

10. Active logs.

11. Actual results.

II.4. Summarize and Report II.4. Summarize and Report Testing ResultsTesting Results

Recording the results of analysis.

Reporting the analysis to involved parties.

Serving as barometer of effectiveness for the test plan.

Could be produced at various levels (system to unit).

Get OrganizedGet Organized

Project procedure

Specific assignments of responsibility

Communication medium

Project handbook

Support library

Project library

III. Software Quality Assurance III. Software Quality Assurance (SQA)(SQA)

III.1 Erroneous Assumptions

III.2 What’s Software Quality Assurance?

III.3 System Quality Dimensions

III.4 Steps for Installing a Basic SQA

III.5 Metrics

III.6 Total Quality Management (TQM)

III.1 Erroneous Assumptions III.1 Erroneous Assumptions About QualityAbout Quality

Quality is intangible and not measurable.

Quality products are too expensive.

Quality problems originate with programmers.

Quality comes from the quality control department.

III.2 What’s Software Quality III.2 What’s Software Quality Assurance (SQA)?Assurance (SQA)?

Software quality is conformance to customer

requirements when such requirements have been

explicitly documented as a foundation for software

development (Functional issues).

Software quality is conformance to a set of implicit

requirements that are desired for all computer-based system. Implicit requirements include testability, maintainability, reliability efficiency, transportability, and security. (Design for internal issues)

What’s Software Quality What’s Software Quality Assurance (SQA)?Assurance (SQA)?

Software quality assurance is a set of procedures that are applied throughout the software engineering process. In essence, the SQA organization assures that standards are properly applied; reviews are conducted; testing is through; changes are properly tracked and recorded; and overall documentation and record keeping are appropriate.

III.3 System Quality DimensionsIII.3 System Quality Dimensions

Three dimensions to system quality.

Quality of design: design it right.

Quality of conformance: build it right.

Quality of performance: it has to work right.

Quality of DesignQuality of Design

Three aspects: Functionality- from the non-technical perspective

Works right; easy to use; within schedule and budget; understandably documented.

Engineering-from the technical perspective

Runs efficiently; conforms to specs; uses the latest techniques and technology.

Adaptability

Flexibility; reusability; maintainability.

Quality of Conformance & Quality Quality of Conformance & Quality of Performanceof Performance

Quality of Conformance

Conformance to design specifications (which relates to the specific applications).

Conformance to standards and conventions (which relates to all applications).

Quality of Performance Uses resources efficiently.

III.4 Steps for Installing a Basic III.4 Steps for Installing a Basic SQASQA

1. Develop a set of software engineering guidelines

Software engineering standards and procedures range from simple informal guidelines to mandated procedural requirements enforced by government edict. These standards can be modified for local needs.

SQA Standards:

DOD-STD-2167; FAA-STD-018; IEEE-STD-8200.1

Steps for Installing a Basic SQASteps for Installing a Basic SQA2. Establish formal technical review guidelines Review the product, not the producer. Set an agenda and maintain it. Limit debate and rebuttal. Enumerate problem areas, but do not attempt to solve every

problem that is noted. Take written notes. Limit the participants and insist upon advance preparation. Develop a checklist for each product that is likely to be

reviewed. Allocate resources and time resources Conduct meaningful training for all reviewers.

Steps for Installing a Basic SQASteps for Installing a Basic SQA

3. Define a testing strategy and distinct black box and white box testing methods.

Black box testing method focuses on the functional requirements, and attempts to find errors in the following categories:

A. Incorrect and missing functions.

B. Interface errors.

C. Errors in data structures

D. Performance errors.


White box testing method focuses on the close examination of procedural detail (on the module’s internal workings). Four traditional criteria: A. Path testing where each executable path through a program

is tested at least once.B. Branch testing which exercises all outcomes of each predicate.C. Structured testing where loops are tested to upper bounds of

iterations.D. Special values testing where ‘singularities’ and boundaries in

the input space are explored and potential exceptions or end-point conditions in the code are targeted at by the choice of input data.


4. Establish a set of procedures for change control.

Establish a set of procedures for documenting and

evaluating the effect of these changes.

Prior to freeze-not subject to managerial approval

After freeze-some formal change evaluation procedure

should exist.

The activity of change control is part of the general

process of software configuration management.

III.5 III.5 MetricsMetrics

It refers to any measurement related to software development. Measurements motivate a organization to improve its process. Useful Metrics:

a. Size: total lines code written; total comment lines; total function points.

b. Productivity: Number of work hours spent on the project; number of work hours spent in each routine.

c. Maintainability:Line of code in each routine; number of decision points in each routine.

d. Overall Quality: Total number of defects; Average defects per thousand of line of codes

III.6 Total Quality Management III.6 Total Quality Management (TQM)(TQM)

It is a conception of quality as an evolutionary process toward perfection instead of conceiving quality as controlling the number of defective products produced.

For TQM to become a reality, the full organizational support of management and early commitment to quality from the analyst and from the business are necessary.

One of the strongest quality assurance actions the system analysis team can take is:

Structured Walkthrough—to use peer reviewers to monitor the system’s programming and overall development.

Quality System DesignQuality System Design

The bottom-up Approach; The top-down approach; and Modular programming concept.

Modules falls into three classes: 1. Control modules- determining the overall program

logic 2. Transformational modules – changing input into output

3. Specialized modules—performing detailed, functional work.

Quality System Design-Quality System Design-(continued)(continued)

Modularity in the Windows Environment:

Dynamic Data Exchange (DDE) updates data in one program based data in another program.

Object Linking and Embedding (OLE) is where an object in a second program retains the properties of an object in the first program.

Object Linking and Embedding Object Linking and Embedding for Databases (OLE DB)for Databases (OLE DB)

Microsoft has defined set of data objects, collectively known as OLE DB.

Allows OLE-oriented applications to share and manipulate sets of data as objects.

OLE DB is an object-oriented specification based on C++ API.

Components can be treated as data consumers and data providers. Consumers take data from OLE DB interfaces and providers expose OLE DB interfaces.

OLE DBOLE DB

Common Gateway Interface (CGI)Common Gateway Interface (CGI)

Specification for transferring information between a Web server and a CGI program.

Server only intelligent enough to send documents and to tell browser what kind of document it is.

But server also knows how to launch other programs. When server sees that URL points to a program

(script), it executes script and sends back script’s output to browser as if it were a file.


CGI defines how scripts communicate with Web servers.

A CGI script is any script designed to accept and return data that conforms to the CGI specification.

Before server launches script, prepares number of environment variables representing current state of the server, who is requesting the information, and so on.

Script picks this up and reads STDIN.


Then performs necessary processing and writes its output to STDOUT.

Script responsible for sending MIME header, which allows browser to differentiate between components.

CGI scripts can be written in almost any language, provided it supports reading and writing of an operating system’s environment variables.


Four primary methods for passing information from browser to a CGI script:

Passing parameters on the command line. Passing environment variables to CGI programs. Passing data to CGI programs via standard input. Using extra path information.

IV. Quality MeasurementIV. Quality Measurement

IV.1 Quality Control & Testing SDLC

IV.2 Quality Metrics

IV.3 SEI Process Maturity Model

IV.1. Quality Control & Testing IV.1. Quality Control & Testing SDLCSDLC

The purpose of testing is to measure quality.

Good testing is a prerequisite to quality control.

Quality control defines what the requirements are (quality is); involves measuring quality; and acts on the results.

Achieving significant testing improvements depends on successfully imposing a new testing life cycle on top of the development cycle.

IV.2. Quality MetricsIV.2. Quality Metrics

The term “Metrics” refers to any measurement related to Software development.Two reasons to measure your process: Any way of measuring the process is superior to not

measuring it at all. Statements like “This new method seems more productive” aren’t good enough.

To argue against metrics is to argue that it’s better not to know what’s really happening on your project.

Useful MetricsUseful Metrics

Size

Total lines of code written

Total comment lines

Total data declarations

Total blank lines

Defect Tracking

Work hours spent on correcting each defect.

Average time required to find a defect.

Quality MetricsQuality Metrics

Metrics Quality Use

Product defect rate

# of defects

----------------

Project size

Quality of product

Cost performance

Total proj cost

-----------------

Project size

Function point


Select metrics to manage.

Select measures to support the metrics.

Function points

Time

Defects

Costs

Metrics must be clearly defined.


schedule = total on-time schedules*

-----------------------------

total schedules**

* sum of al function points active on-time projects times

The number of elapsed days to date in each on time

project.

** sum of all function points in active projects times the number of days in each elapsed projects.

IV.3 Process Maturity ModelIV.3 Process Maturity ModelSEI (Software Engineering Institute) funded by DOD to study software development and disseminate information about effective development practices. It identifies five levels of software process maturity:1. Anarchy- Operate without formal planning2. Folklore- Tribe Knowledge3. Standards-Methodology written down, but no one has collected data to measure its effectiveness4. Measurements- Data is collected to assess the process

effectiveness.5. Optimization- An organization focuses repeatability and

measurement primarily to improve product quality.

Process Maturity ModelProcess Maturity Model

Three factors are needed to improve process

maturity:

1. Repeatability (Methodology)

2. Measurability (Metrics)

3. Feedback (Optimization)

Repeatability (Methodology)Repeatability (Methodology)

A process is made repeatable by the use of a standard methodology, thus laying the groundwork for the measurement at the higher levels.

Feedback (Optimization)Feedback (Optimization)

The salient characteristics of the SEI organizational maturity model at its highest maturity are repeatability, measurability, and controlled variations and feedback.

V. Concluding Remarks- Elements V. Concluding Remarks- Elements of A Software Quality Programof A Software Quality Program

Software quality objectives: external vs. internal

characteristics. Explicit quality activities: no more quick and dirty

programming. Testing strategy. Software engineering guidelines. Informal technical reviews: desk checking Formal technical reviews: a peer review, a

walkthrough

Elements of A Software Quality Elements of A Software Quality ProgramProgram

External audits: audits outside the organization. Development processes: effective risk management. Change control procedure. Measurements of results.

Quality ActivitiesQuality Activities

Requirements

Design

Code construction

System implementation Maintenance

Verify requirements; Test Strategy; Test Plan Verify design; prepare

test cases; place cases under CM

Verify code; create test cases; execute testing Alpha test; beta test Regression test; manage

change control

System Phase Activity

chapter 12: software testing & quality assurance instructor: paul k chen

Documents