contents quality assurance what is quality assurance

PVK-HT02 [email protected] 1

ContentsØ Introduction 4

Ø RequirementsEngineering 4

Ø Project Management 4

Ø Software Design 4

Ø Detailed Design andCoding 4

ØQuality AssuranceØ Maintenance

Analysis

Design

Testing

Coding

Operation andMaintenance

Installation

Require-ments

Requirements

Specification

Planning


Quality Assurance

q Introductionq Testing

ØTesting Phases and ApproachesØBlack-box TestingØWhite-box TestingØSystem Testing


What is Quality Assurance?

q Constructive vs analytic approaches to QAq Qualitative vs quantitative quality standardsq Measurement

o Derive qualitative factors from measurablequantitative factors

Ł Software Metrics

QA is the combination of planned and unplannedactivities to ensure the fulfillment of predefined qualitystandards.


Approaches to QAq Constructive Approaches

o Syntax-directed editorso Type systemso Transformational programmingo Coding guidelineso ...

q Analytic approaches

o Inspectionso Static analysis tools (e.g. lint)o Testingo ...

Usage of methods, languages, and tools that ensure thefulfillment of some quality factors.

Usage of methods, languages, and tools to observe thecurrent quality level.


Fault vs Failure

?!human error fault failure

can lead to can lead to

q Different types of faultsŁ Different identification techniquesŁ Different testing techniques

Ł Fault prevention and -detection strategiesshould be based on expected fault profile


Specification/requirements

Environment/support

Documen-tation OtherDesign Code

Fault origin: WHERE?

Missing Unclear Wrong Changed Better way

Fault mode: WHY?

Fau

lttype:WHAT? Requirements

orspecifications

Functionality

HW interface

SW interface

User interface

Functionaldescription

Test HW

Test SW

Integration SW

Developmenttools

Logic

Computation

Data handling

Moduleinterface/

implementation

Standards

(Inter-)Processcommunications

Data definition

Module design

Logicdescription

Error checking

Standards

HP´s Fault Classification


Fault Profile of a HP Division

6%

19%

5%

4%

5%

32%

18%

11%

Data handling

Documentation

Requirements

Hardware

Process/interprocess

Logic

Computation

Other codeSee [Pfleeger 98].


Unittest

Unittest

Unittest

Integrationtest

Functiontest

Performancetest

Acceptance(α,β) test

Installationtest

Com

pone

ntco

deC

ompo

nent

code

.

.

.

Testedcom

ponent

Test

edco

mpo

nent

Integratedmodules

Functioningsystem

Verifiedsoftware

Accepted,validatedsystem

SYSTEMIN USE!

Designspecifications

Systemfunctionalrequirements

Othersoftware

requirements

Customerrequirementsspecification

Userenvironment

Testing Phases

Systemtest

Pre-implementationtest


Pre-Implementation Testingq Inspections

o evaluation of documents and code prior to technicalreview or testing

q Walkthrougho In teamso Examine source code/detailed design

q Reviewso More informalo Often done by document owners

q Advantageso Effectiveo High learning effecto Distributing system knowledge

q Disadvantageso Expensive


Testing vs “Proofing”Correctness

q Verificationo Check the design/code against the requirementsŁ Are we building the product right?

q Validationo Check the product against the expectations of the customerŁ Are we building the right product?

q Testing

Ł Testing can neither proof that a program is error free, nor thatit is correct

Testing is the process in which a (probably unfinished)program is executed with the goal to find errors.

[Myers 76]

Testing can only show the presence of errors, never theirabsence.

[Dijkstra 69]


Goals of Testing

q Detect deviations from specificationso Debuggingo Regression testing

q Establish confidence in softwareo Operational testing

q Evaluate properties of softwareo Reliabilityo Performanceo Memory use/leakageo Securityo Usability PVK-HT02 [email protected] 15

Principles of SoftwareTesting*

q Complete testing is not possibleq Testing is creative and difficultq A major objective of testing is defect

preventionq Testing must be risk-basedq Testing must be plannedq Testing requires independence* Hetzel, The Complete Guide to Software Testing


Fundamental Steps of SoftwareTesting

q Understand requirements andspecifications

q Create the execution environmentq Select test casesq Execute & evaluate test casesq Evaluate test progressq Information feedback & feedforward


Unit Testing

Defn: Tests the smallest individually executablecode units.

Objective: Find faults in the units. Assurecorrect functional behavior of units.

By: Usually programmers.Tools:q Test driver/harnessq Coverage evaluatorq Automatic test generator


Test Methodsq Structural testing (white-box, glass-box)

o Uses code/detailed design is to develop test caseso Typically used in unit testingo Approaches:

• Coverage-based testing• Symbolic execution• Data flow analysis• ...

q Functional testing (black-box)o Uses function specifications to develop test caseso Typically used in system testingo Approaches:

• Equivalence partitioning• Border case analysis• ...

time

developblack-boxtest cases

developwhite-boxtest cases

performwhite-box

testing

performblack-box

testing


Test Preparation

q Exhaustive testing is prohibited, because ofthe combinatorial explosion of test cases

Ł Choose representative test data

for i := 1 to 100 doif a = b then

Xelse

Y;

i paths to test #tests

1 X, Y 22 XX, XY, YX, YY 43 XXX, XXY, ... 8

...100 2100 2 ∗ 2100 - 2 > 2,5 ∗ 1030

With 106 tests/sec this would take 8∗ 1016 yearsChoose test data (test cases)


Test Case Developmentq Problems:

o Systematic way to develop test caseso Find a satisfying set of test cases

q Test case ≠≠≠≠ test dataq Test data: Inputs devised to test the systemq Test case:

o Situation to testo Inputs to test this situationo Expected outputsŁ Test are reproducible


Black-box Testing

q Test generation without knowledge ofsoftware structure

q Also called specification-based orfunctional testing

Ł Equivalence partitioningŁ Boundary-value analysis


Equivalence Partitioning

System

Input data

Output data

Inputs causinganomalousbehaviour

Outputs whichrevealthe presence offaults

Input- and output data canbe grouped into classeswhere all members in theclass behave in acomparable way.

Ł Define the classesŁ Choose representatives

u Typical elementu Borderline cases

x ∈ [25 .. 100]class 1: x < 25class 2: x >= 25 and x <= 100class 3: x > 100


White-box Testing

Methods based on internal structure ofcode

q Statement coverageq Branch coverageq Path coverageq Data-flow coverage

if a thenb

elsec;

d

while a dob;

c

a

b

d

c

b

c

a


Statement coverage

1

3

2 4

5

6 7

8

2 test cases: 12467; 13567

q Every statement is at least executedonce in some test


Branch Coverage

1

3

2 4

5

6 7

8

2 test cases: 12467; 1358

q For every decision point in the graph, eachbranch is at least chosen once


Condition Coverage

q Test all combinations of conditions inboolean expressions at least once

Ł Why in boolean expressions only?

if (X or not (Y and Z) and ... thenb;

c := (d + e * f - g) div op( h, i, j);


Path Coverage

Assure that all paths in the control-flowgraph are executed.

What is the definition of all paths?q All loop-free pathsq All loop-free paths, plus all n-iterations

of loopsq A set of basis paths for the graph.


Path Coverage

1

3

2 4

5

6 7

8


Path Coverage

1

3

2 4

5

6 7

8

4 test cases: 12467; 1358; 1248; 13567


Data-flow testing

Def of a variable v – assignment of value to vUse of variable v – access of the value of vA def-use association for variable v is a def, a

use and a path from the def to the use whichcontains no intervening definition of v.

All-defs – For each def d, test at least one pathfrom d to some use of d

All-uses – For each def d, and for each use ui ofd, test at least one path from d to ui


Data Flow CoverageAll-uses coverage

x :=2

x :=1

z := 2*r

x :=3

z := 2*x-y

z :=x+y

y :=2r :=4

Red path covers the defs y :=2; r :=4; x :=1

Blue path covers y :=2; x :=3. Does notcover x :=2


Coverage-based Testingq Advantages

o Systematic way to develop test caseso Simple model of underlying programo Measurable results (the coverage)o Extensive tool support

• Flow graph generators• Test data generators• Bookkeeping• Documentation support

q Disadvantageso Code must be availableo Does not (yet) work well for data-driven programs


Integration TestingDefn: Testing two or more units or componentsObjectives

o Interface errorso Functionality of combined units; look for

problems with functional threadsBy: Developers or Testing groupTools: Interface analysis; call pairsIssues:q Strategy for combining unitsq Assuring compatibility and correctness of

externally-supplied components


Integration TestingHow to integrate & test the systemq Top-downq Bottom-upq Critical units firstq Functionality-oriented (threads)Implications of build orderq Top-down => stubs; more thorough top-levelq Bottom-up => drivers; more thorough bottom-

levelq Critical => stubs & drivers.

A

DCB

GFE Test E Test F Test G

Test B,E,F Test C Test D,G

Test all


System Testing

Defn: Test the functionality, performance,reliability, security of the entire system.

By: Separate test group.Objective: Find errors in the overall system

behavior. Establish confidence in systemfunctionality. Validate that system achieves itsdesired non-functional attributes.

Tools: User simulator. Load simulator


Realities of System Testing

q Available time for testing is shorto Compressing development risks introducing problemso Compressing testing risks missing critical problems

q Testers want to start testing earlyq System testing requires an available systemq Developers resist testing until system is “ready”To optimize use of the existing resources, use riskanalysis.


Acceptance Testing

Defn: Operate system in user environment, withstandard user input scenarios.

By: End userObjective: Evaluate whether system meets

customer criteria. Determine if customer willaccept system.

Tools: User simulator. Customer testscripts/logs from operation of previoussystem.


Regression Testing

Defn: Test of modified versions of previouslyvalidated system.

By: System or regression test group.Objective: Assure that changes to system have

not introduced new errors.Tools: Regression test base, capture/replayIssues: Minimal regression suite, test

prioritization


Test Automation

Automation has several meaningsq Test generation: Produce test cases by

processing of specifications, code, model.q Test execution: Run large numbers of test

cases/suites without human intervention.q Test management: Log test cases & results;

map tests to requirements & functionality;track test progress & completeness


Issues of Test AutomationAutomating Test Executionq Does the payoff from test automation justify the

expense and effort of automation?q Learning to use the automation tool is non-trivialq Testers become programmersq All tests, including automated tests, have a finite

lifetime.q Complete automated execution implies putting

the system into the proper state, supplying theinputs, running the test case, collecting theresult, verifying the result.


Best uses of automated testexecution

q Load/stress tests--Nearly impossible to have1000 human testers simultaneouslyaccessing a database.

q Regression test suites--Tests collected fromprevious releases; run to check that updatesdon’t break previously correct operation.

q Sanity tests (smoke tests)--run after everynew system build to check for obviousproblems.

Test documentation

q Test plan: describes system and plan forexercising all functions and characteristics

q Test specification and evaluation: detailseach test and defines criteria for evaluatingeach feature

q Test description: test data and proceduresfor each test

q Test analysis report: results of each test


The Key Problems of SoftwareTesting

q Selecting or generating the right testcases.

q Knowing when a system has beentested enough.

q Knowing what has beendiscovered/demonstrated by executionof a test suite.

contents quality assurance what is quality assurance

Documents