iseb software testing presentation

8/8/2019 ISEB Software Testing Presentation

1/414

SIM Group Ltd., SQS Group AG, 2002

ISEB Foundation Certificate in Software Testing

Testing TerminologyTesting Terminology


2/414


Testing Terminology

IT is an industry riddled with mysterious words, terms andIT is an industry riddled with mysterious words, terms andTLATLAss

Software testing is no exceptionSoftware testing is no exception

Historically there has not been a generally accepted set ofHistorically there has not been a generally accepted set of

testing definitionstesting definitions


3/414


Testing Terminology

To resolve this issue, the BCS SIGIST created a StandardTo resolve this issue, the BCS SIGIST created a StandardGlossary of Testing TermsGlossary of Testing Terms

British Computer SocietyBritish Computer Society

Special Interest Group In Software TestingSpecial Interest Group In Software Testing

This is now a British StandardThis is now a British Standard

BS7925BS7925--11


4/414SIM Group Ltd., SQS Group AG, 2002

Testing Terminology

You will need to read the document and start to learn theYou will need to read the document and start to learn theterminology used in software testingterminology used in software testing

You will be exposed to the terminology and customers willYou will be exposed to the terminology and customers willassume you know what they meanassume you know what they mean




Why Testing is NecessaryWhy Testing is Necessary



Why Testing is Necessary

In this session we willIn this session we will

Understand what testing is and why it is necessaryUnderstand what testing is and why it is necessary

Define some of the words used in software testingDefine some of the words used in software testing

Look at the cause and cost of errorsLook at the cause and cost of errors




Why Test?Why Test?

To find bugsTo find bugs

What is a bug?What is a bug?


8/414



What is Testing?What is Testing?

the process of executing a program with the intent to certify itthe process of executing a program with the intent to certify itss

QualityQuality

MillsMills

the process of executing a program with the intent of findingthe process of executing a program with the intent of finding

failures / faultsfailures / faults

MyersMyers

the process of exercising software to detect bugs & to verify ththe process of exercising software to detect bugs & to verify thatat

it satisfies specified functional & nonit satisfies specified functional & non--functional requirementsfunctional requirements


9/414



Why is Testing becoming more important?Why is Testing becoming more important?

The Y2K problemThe Y2K problem

EMUEMU

e Commercee Commerce

Increased user baseIncreased user base

Increased complexityIncreased complexity

Speed to MarketSpeed to Market


10/414


Definitions


11/414


12/414


Definitions

FAULTFAULT

A Fault, if encountered, may cause a failureA Fault, if encountered, may cause a failure

FAILUREFAILUREA deviation of the software from its expected delivery orA deviation of the software from its expected delivery or

serviceservice


13/414


Definitions

DEFECTDEFECT

The departure of a quality characteristic from its specified valThe departure of a quality characteristic from its specified valueue

that results in a product or service not satisfying its normalthat results in a product or service not satisfying its normal

usage requirementsusage requirements


14/414


Definitions

RELIABILITYRELIABILITY

The probability that software will not cause the failure of aThe probability that software will not cause the failure of a

system for a specified period of time under specifiedsystem for a specified period of time under specified

conditionsconditions


15/414


Definitions

QUALITYITY

The totality of the characteristics of an entity that bear on itThe totality of the characteristics of an entity that bear on itss

ability to satisfy stated or implied needsability to satisfy stated or implied needs

T i & Q li


16/414


Testing & Quality

Does testing improve quality?Does testing improve quality?

Testing does not build Quality into the softwareTesting does not build Quality into the software

Testing is a means of determining the Quality of the SoftwareTesting is a means of determining the Quality of the Software

under Testunder Test

D fi iti


17/414


Definitions

QUALITY ASSURANCEQUALITY ASSURANCE

All those planned actions used to fulfil the requirements forAll those planned actions used to fulfil the requirements for

qualityquality

Wh d ?


18/414


Why do errors occur?

Wh do errors occ r?


19/414



How Errors OccurHow Errors Occur

No one is perfect!No one is perfect!

We all make mistakes or omissionsWe all make mistakes or omissions

The more pressure we are under the more likely we are toThe more pressure we are under the more likely we are to

make mistakesmake mistakes

In IT development we have time and budgetary deadlines toIn IT development we have time and budgetary deadlines to

meetmeet

Poor TrainingPoor Training



20/414



How Errors OccurHow Errors Occur

Poor CommunicationPoor Communication

Requirements not clearly definedRequirements not clearly defined

Requirements change & are not properly documentedRequirements change & are not properly documented

Data specifications not completeData specifications not complete

ASSUMPTIONS!ASSUMPTIONS!



21/414



The Cost Of ErrorsThe Cost Of Errors

A single failure may incur little costA single failure may incur little cost -- or millionsor millions

Report layouts may be wrongReport layouts may be wrong -- little true costlittle true cost

Or a significant error may cost millionsOr a significant error may cost millions

ArianeAriane V, Venus Probe, Mars Explorer and Polar LanderV, Venus Probe, Mars Explorer and Polar Lander



22/414




In extreme cases a software or systems error may costIn extreme cases a software or systems error may cost

LIVESLIVES

Usually safety critical systems are tested exhaustivelyUsually safety critical systems are tested exhaustively

Aeroplane, railway, nuclear power systems etcAeroplane, railway, nuclear power systems etc

Unfortunately there are exceptions to thisUnfortunately there are exceptions to this

London Ambulance ServiceLondon Ambulance Service



23/414




The cost of defects increases proportionately (tenfold?) withThe cost of defects increases proportionately (tenfold?) with

the passing of each successive stage in the systemthe passing of each successive stage in the systemdevelopment process before they are detecteddevelopment process before they are detected

To correct a problem at requirements stage may costTo correct a problem at requirements stage may cost 11

To correct the problem postTo correct the problem post--implementation may costimplementation may cost 000000ss



24/414





25/414



Why Test?Why Test?

Identifies faultsIdentifies faults

Reduces live defectsReduces live defects

Improves the quality of the users applicationImproves the quality of the users application

Increases reliabilityIncreases reliability

To help ensure live failures donTo help ensure live failures dont impact costs & profitabilityt impact costs & profitability

To help ensure requirements are satisfiedTo help ensure requirements are satisfied

To ensure legal requirements are metTo ensure legal requirements are met

To help maintain the organisationTo help maintain the organisations reputations reputation

Provides a measure of qualityProvides a measure of quality



26/414



Quality MeasurementsQuality Measurements

Quality can be measured by testing forQuality can be measured by testing for

correctnesscorrectness

reliabilityreliability

usabilityusabilitymaintainabilitymaintainability

testabilitytestability

reusabilityreusability



27/414



How much is enough?How much is enough?

How do we know when to stop?How do we know when to stop?

There are many factors to considerThere are many factors to consider



28/414



Exhaustive TestingExhaustive Testing

Find all faults by testing everything?Find all faults by testing everything?

To test everything is rarely possibleTo test everything is rarely possible

The time required makes it impracticalThe time required makes it impractical

The resource commitment required makes it impracticalThe resource commitment required makes it impractical



29/414


y est g s ecessa y

If we canIf we cant test everything, what can we do?t test everything, what can we do?

Managing and reducing RiskManaging and reducing Risk

Carry out a Risk Analysis of the applicationCarry out a Risk Analysis of the application

Prioritise tests to focus on the main areas of riskPrioritise tests to focus on the main areas of risk

Apportion time relative to the degree of risk involvedApportion time relative to the degree of risk involved

Understand the risk to the business of the software notUnderstand the risk to the business of the software not

functioning correctlyfunctioning correctly



30/414


y g y

Should you stop testing when the product goes live?Should you stop testing when the product goes live?

Continuing testing beyond the implementation date should beContinuing testing beyond the implementation date should be

consideredconsidered

Better that you find the errors than the usersBetter that you find the errors than the users



31/414


y g y

SummarySummary

The purpose of testing is to find faultsThe purpose of testing is to find faults

Faults can be fixed, making better softwareFaults can be fixed, making better softwareBetter software is more reliable, less prone to failuresBetter software is more reliable, less prone to failures

Establish the relationship between the software and itsEstablish the relationship between the software and its

specificationspecification

Testing enables us to measure the quality of the softwareTesting enables us to measure the quality of the software

This enables us to understand & manage the risk to theThis enables us to understand & manage the risk to the

businessbusiness


32/414



The Test ProcessThe Test Process

The Test Process


33/414



Look at the steps involved in the test processLook at the steps involved in the test process

Look in detail at each stepLook in detail at each step

The Test Process


34/414


What is the objective of a test?What is the objective of a test?

AA successfulsuccessful test is one thattest is one that doesdoes detect a faultdetect a fault

The Test Process


35/414


COMPLETION

E

XECUTIO

N

TEST MANAGEMENT

TEST ENVIRONMENT MANAGEMENT

TEST ASSET MANAGEMENT

R

ECORDIN

G

PLANNING

SPECIFICAT

ION

The Test Process


36/414


Five steps in the Test Process areFive steps in the Test Process are

Test PlanningTest Planning

Test Analysis and SpecificationTest Analysis and Specification

Test ExecutionTest Execution

Test Recording (Verification)Test Recording (Verification)

Checking for CompletionChecking for Completion

Test Planning


37/414


The Test Plan describes howThe Test Plan describes howthe Test Strategy isthe Test Strategy is

implementedimplemented

Test Planning


38/414


Contents of a Test Plan includeContents of a Test Plan include

BackgroundBackground

Reference documentsReference documents ApproachApproach

MethodMethod

TimetableTimetable

ResourcesResources

DependenciesDependencies

ReportingReporting

Test Asset IdentificationTest Asset Identification

Exit CriteriaExit Criteria

Test Planning


39/414


The most critical stage of theThe most critical stage of theprocessprocess

Effort spent now will beEffort spent now will be

rewarded laterrewarded later

The foundation on whichThe foundation on which

testing is builttesting is built

Test Specification


40/414


Three step processThree step process

Preparation & analysisPreparation & analysis

Building test casesBuilding test cases

Define expected resultsDefine expected results

Test Specification


41/414


Test preparationTest preparation

Analyse the ApplicationAnalyse the Application

Identify good test conditionsIdentify good test conditions

Identify test casesIdentify test cases

Document thoroughlyDocument thoroughly

CrossCross--referencingreferencing

Test Specification


42/414


Build Test CasesBuild Test Cases

Test cases comprise:Test cases comprise:

Standing DataStanding Data

Transaction DataTransaction Data

ActionsActions

Expected ResultsExpected Results

Test Specification


43/414


Expected ResultsExpected Results

The outcome of each actionThe outcome of each action

The state of the applicationThe state of the application

during & afterduring & after

The state of the data duringThe state of the data during& after& after

Test Specification


44/414


CrossCross--Referencing &Referencing &ClassificationClassification

Enables maintainability ofEnables maintainability of

Test AssetsTest Assets

Allows testing to beAllows testing to be

performed in a focussedperformed in a focussedmanner directed at specificmanner directed at specific

areasareas

Test Execution


45/414


Test Execution checklistTest Execution checklist

Test execution schedule / logTest execution schedule / log

Identify which tests are to be runIdentify which tests are to be run

Test environment primed & readyTest environment primed & ready

Resources ready, willing & ableResources ready, willing & able BackBack--up & recovery procedures in placeup & recovery procedures in place

Batch runs planned & scheduledBatch runs planned & scheduled

Then we are ready to run the testsThen we are ready to run the tests

Test Recording


46/414


Test verificationTest verification

If planning and preparation is sufficiently detailed this is theIf planning and preparation is sufficiently detailed this is the

easy part of testingeasy part of testing

The test is run to verify the application under testThe test is run to verify the application under test

The test itself either passes or fails!The test itself either passes or fails!

Test Recording


47/414


The test log should recordThe test log should record

Software and test versionsSoftware and test versions

Specifications used as test baseSpecifications used as test base

Test timingsTest timings

Test resultsTest results Actual resultsActual results

Expected resultsExpected results

Defect details for erroneous testsDefect details for erroneous tests

Test Completion


48/414


Test Exit CriteriaTest Exit Criteria

Used to determine when toUsed to determine when to

implement the softwareimplement the software

Key Functionality testedKey Functionality tested

Test coverageTest coverage

Budget used?Budget used?

Defect detection rateDefect detection rate

Performance satisfactoryPerformance satisfactory

The Test Process


49/414


SummarySummary

There are five steps in the Test ProcessThere are five steps in the Test Process

Test PlanningTest Planning

Test Analysis and SpecificationTest Analysis and Specification

Test ExecutionTest Execution Test Recording (Verification)Test Recording (Verification)

Checking for CompletionChecking for Completion


50/414



The Psychology Of TestingThe Psychology Of Testing

The Psychology Of Testing


51/414



Understand what qualities make good testersUnderstand what qualities make good testers

Look at a testersLook at a testers relationship with developersrelationship with developers

Look at a testersLook at a testers relationship with managementrelationship with management

Understand the issues with testing independenceUnderstand the issues with testing independence

What makes a Tester?


52/414


Testing is primarily to find faultsTesting is primarily to find faults

Can be regarded asCan be regarded as destructivedestructive

Development isDevelopment is constructiveconstructive

Testing asks questionsTesting asks questions

Testers need to ask questionsTesters need to ask questions

A tester needs many qualities...A tester needs many qualities...



53/414


Intellectual qualitiesIntellectual qualities

Can absorb incomplete factsCan absorb incomplete facts

Can work with incomplete factsCan work with incomplete facts

Can learn quickly on many levelsCan learn quickly on many levels

Good verbal communicationGood verbal communication Good written communicationGood written communication

Ability to prioritiseAbility to prioritise

SelfSelf--organisationorganisation


54/414



55/414


More skills to acquireMore skills to acquire

How to find bugsHow to find bugs -- planning, preparation & executionplanning, preparation & execution

How to understand systemsHow to understand systems

How to read specificationsHow to read specifications

How to extract testable functionalityHow to extract testable functionality How to work efficientlyHow to work efficiently

How to focus on essentialsHow to focus on essentials

Reporting Defects


56/414


Defects need to be reported toDefects need to be reported to

Developers to enable them to fix themDevelopers to enable them to fix them

Management so they can track progressManagement so they can track progress

Communication to both groups is vitalCommunication to both groups is vital

Communication with Developers


57/414


A good relationship is vitalA good relationship is vital

Developers need to keep testers up to date with changes toDevelopers need to keep testers up to date with changes to

the applicationthe application

Testers need to inform developers of defects to allow fixes toTesters need to inform developers of defects to allow fixes to

be appliedbe applied

Communication with Management


58/414


Managers need progress reportsManagers need progress reports

The best way is through metricsThe best way is through metrics

Number of tests planned & preparedNumber of tests planned & prepared

Number of tests executed to dateNumber of tests executed to date

Number of defects raised & fixedNumber of defects raised & fixed How long planning, preparation and execution stages takeHow long planning, preparation and execution stages take


59/414


The Psychology of Testing

Testing IndependenceTesting Independence

Testing Independence


60/414


It is important that testing is separate fromIt is important that testing is separate fromdevelopmentdevelopment

The developer is likely to confirm adherence not deviationThe developer is likely to confirm adherence not deviation

The developer will make assumptionsThe developer will make assumptions -- the same whenthe same when

testing as developingtesting as developing

Testing Independence


61/414


Levels of IndependenceLevels of Independence

LowLow -- Developers write their own testsDevelopers write their own tests

MediumMedium -- Tests are written by another developerTests are written by another developer

HighHigh -- Tests written by an independent bodyTests written by an independent body Tests written by another sectionTests written by another section

Tests written by another organisationTests written by another organisation

UtopiaUtopia -- Tests generated automaticallyTests generated automatically

The Psychology Of Testing


62/414


SummarySummary

Testers require a particular set of skillsTesters require a particular set of skills

The desire to break thingsThe desire to break things The desire to explore and experimentThe desire to explore and experiment

CommunicationCommunication

QuestioningQuestioning

Testing requires a different mentality to developmentTesting requires a different mentality to development

DestroyingDestroying things rather than creating themthings rather than creating them

Testing should be separate from developmentTesting should be separate from development


63/414



ReRe--Testing & Regression TestingTesting & Regression Testing

Re-Testing & Regression Testing


64/414



Understand how fault fixing and reUnderstand how fault fixing and re--testing are key to thetesting are key to the

overall testing processoverall testing process

Look at how test repeatability helpsLook at how test repeatability helps

Understand what regression testing is and where it fits inUnderstand what regression testing is and where it fits in

Understand how to select test cases for regression testingUnderstand how to select test cases for regression testing

Re-Testing


65/414


Testing is directed at finding faultsTesting is directed at finding faults

These faults will need to be corrected and a new version ofThese faults will need to be corrected and a new version of

the software releasedthe software released

The tests will need to be run again to prove that the fault isThe tests will need to be run again to prove that the fault is

fixedfixed

This is known as ReThis is known as Re--TestingTesting


66/414

Regression Testing


67/414


ReRe--testing of a previously tested program followingtesting of a previously tested program followingmodification to ensure that FAULTS have not beenmodification to ensure that FAULTS have not been

introduced or uncovered as a result of the changesintroduced or uncovered as a result of the changes

mademade

Regression Testing

Module 93 v1 3 5Module 93 v1.3.5

H fi thiHow can you fix this


68/414


Module 14 v1.4.2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Declare String x =Date();~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Module 14 v1.4.2

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Declare String x =Date();~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Module 41 v1.8.9

~~~~~~~~~~~~~~~~Declare Integer x = 0;~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~For x = 1 to 5~~~~~~~~~~~~~~~~

Module 41 v1.8.9

~~~~~~~~~~~~~~~~

Declare Integer x = 0;~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~For x = 1 to 5~~~~~~~~~~~~~~~~

Module 93 v1.3.5

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Print (x);~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Module 93 v1.3.5

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Print (x);~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Module 14 v1.4.3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Global String x = Date();~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Module 14 v1.4.3

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Global String x = Date();~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

How can you fix thisbut break that?

How can you fix thisbut break that?

Regression Testing

T t ill l d t b h h ki


69/414


Tests will also need to be reTests will also need to be re

--run when checkingrun when checking

software upgradessoftware upgrades

Regression tests should be run whenever there is a changeRegression tests should be run whenever there is a changeto the software or the environmentto the software or the environment

Regression tests are executed to prove aspects of a systemRegression tests are executed to prove aspects of a systemhave not changedhave not changed

Regression Testing is a vital testing techniqueRegression Testing is a vital testing technique


70/414

Regression Testing

Regression Testing is the ideal foundation for


71/414


Regression Testing is the ideal foundation forRegression Testing is the ideal foundation for

AutomationAutomation

Selecting test cases is vital, and requires a degree ofSelecting test cases is vital, and requires a degree ofknowledge of the applications and their expected evolutionknowledge of the applications and their expected evolution

Re-Testing & Regression Testing

Summary


72/414


SummarySummary

Once a fault has been fixed the software MUST be reOnce a fault has been fixed the software MUST be re--testedtested

New faults may have been introduced by the fixNew faults may have been introduced by the fix

Existing faults may have been uncovered by the fixExisting faults may have been uncovered by the fix

Tests need to be written to enable their reTests need to be written to enable their re--useuse

ReRe--testing is the rerunning of failed tests once a fix has beentesting is the rerunning of failed tests once a fix has been

implemented to check that the fix has workedimplemented to check that the fix has worked

Regression testing is running of a wider test suite to check forRegression testing is running of a wider test suite to check for

unexpected errors in unchanged codeunexpected errors in unchanged code


73/414

Expected Results



74/414



Understand the need to define expected resultsUnderstand the need to define expected results

Understand where expected results can be foundUnderstand where expected results can be found

Expected Results

Expected results = expectedExpected results = expected


75/414


Expected results = expectedExpected results expected

outcomesoutcomes

Identifying required behaviourIdentifying required behaviour

If the expected outcome of aIf the expected outcome of a

test is not defined the actualtest is not defined the actual

output may be misinterpretedoutput may be misinterpreted

Expected Results

Running a test with only aRunning a test with only a


76/414


Running a test with only ag y

general concept of thegeneral concept of the

outcome is fataloutcome is fatal

Expected Results

It is vital the Expected ResultsIt is vital the Expected Results


77/414


pp

are defined with the tests,are defined with the tests,

before they are usedbefore they are used

You cannot decide whether aYou cannot decide whether a

test has passed just becausetest has passed just because

it looks rightit looks right


78/414

Expected Results

Consult the Oracle!Consult the Oracle!


79/414


Expected Results

SummarySummary


80/414


Su a yy

Without defining expected results how do you know if a testWithout defining expected results how do you know if a test

has passed or failed?has passed or failed?

Expected results can be found in the system specification andExpected results can be found in the system specification and

by asking experienced business usersby asking experienced business users


81/414



Prioritisation of TestsPrioritisation of Tests

Prioritisation of Tests



82/414


Understand why we need to prioritise testsUnderstand why we need to prioritise tests

Understand how we decide the priority of individual testsUnderstand how we decide the priority of individual tests

Prioritisation of tests

It is not possible to testIt is not possible to test


83/414


everythingeverything

Therefore faults will getTherefore faults will get

through to the live systemthrough to the live system

We must do the best testingWe must do the best testing

possible in the time availablepossible in the time available

This means we mustThis means we must

prioritise and focus testingprioritise and focus testing

on the prioritieson the priorities

Aspects to Consider

SeveritySeverity Frequency of ChangeFrequency of Change


84/414


ProbabilityProbability

VisibilityVisibility

Priority of RequirementsPriority of Requirements

Customer RequirementsCustomer Requirements

Vulnerability to errorVulnerability to error

Technical criticalityTechnical criticality

ComplexityComplexity

Resource availabilityResource availability

Business Criticality

What elements of theWhat elements of the


85/414


application are essential toapplication are essential to

the success of thethe success of the

organisation?organisation?


86/414

Technical Factors

How complex is it?How complex is it?


87/414


How likely is an error?How likely is an error?

How often does this change?How often does this change?

Prioritisation of Tests

SummarySummary


88/414


Due to the last minute nature of testing, it is necessary toDue to the last minute nature of testing, it is necessary to

prioritise testsprioritise tests

There will never be enough time to complete all testsThere will never be enough time to complete all tests

Therefore those tests that cover those areas deemed mostTherefore those tests that cover those areas deemed most

important (to the business, highest risk etc) must be testedimportant (to the business, highest risk etc) must be tested

firstfirst


89/414

Models for Testing



90/414


Look at two of the most commonly accepted testing modelsLook at two of the most commonly accepted testing models


91/414


92/414


93/414

A simple V-Model

high level


94/414


low level time

requirements

design

code unittest

system

test

UAT

TT

TT

TT


95/414

Models for Testing

V-Model - where do we start testing?


96/414


level

time

RequirementsRequirements

DesignDesign

CodeCode Unit TestUnit Test

System TestSystem Test

UATUATTT

TT

TT

Models for Testing

V-Model - Start Testing Early


97/414


Reqs

level

time

TT

TT

TT

TT

TT

RR

RR

RR

RR

RR

PP

PP

RRPP TT

Spec

Code

UAT

System Test

Unit Test

Models for Testing

SummarySummary


98/414


The VThe V--Model is the most common approaches model forModel is the most common approaches model for

testingtesting

There are other models used for testing, but these are lessThere are other models used for testing, but these are less

widely usedwidely used

Inc. V,V & TInc. V,V & T

Other models have been createdOther models have been created


99/414



Economics of TestingEconomics of Testing


100/414

Economics of Testing

The earlier a fault is found, the cheaper it is to remedyThe earlier a fault is found, the cheaper it is to remedy


101/414


Faults in the requirements can lead to major reFaults in the requirements can lead to major re--engineering ofengineering of

the entire systemthe entire system

Many faults can be found using reviewsMany faults can be found using reviews

Reviews of documentation and / or codeReviews of documentation and / or code

Allows testing to find moreAllows testing to find more substantialsubstantial faultsfaults


102/414


In initial testing, defect fix times will average betweenIn initial testing, defect fix times will average between

10 to 20 minutes10 to 20 minutes


103/414


10 to 20 minutes10 to 20 minutes

In integration testing each defect can cost an hour orIn integration testing each defect can cost an hour ormoremore

In system test each defect can cost 10 to 40 or moreIn system test each defect can cost 10 to 40 or more

engineer hoursengineer hours

Watts HumphreyWatts Humphrey


Early test design can prevent fault multiplicationEarly test design can prevent fault multiplication


104/414


Analysis of specification during test preparation often bringsAnalysis of specification during test preparation often brings

faults in the specification to lightfaults in the specification to light

If faults in documentation are not found then the system mayIf faults in documentation are not found then the system may

be developed incorrectlybe developed incorrectly


105/414


Economics of testing

An exampleAn example


Finding the error in the requirementsFinding the error in the requirements


106/414


Change requirements documentChange requirements document


Finding the error in the specificationsFinding the error in the specifications


107/414



Change specificationsChange specifications


108/414


Finding the error in system testingFinding the error in system testing


109/414




Change the codeChange the code

Change the tests, reChange the tests, re--test & regression testtest & regression test


Finding the error in UATFinding the error in UAT


110/414






ReRe--do UATdo UAT


Finding the error in LiveFinding the error in Live

Ch i t d tChange req irements doc ment


111/414






ReRe--do UATdo UAT

Plus...Plus...


Finding the error in LiveFinding the error in Live


112/414


User to report and log faultUser to report and log fault

Users must be advised of fix, workUsers must be advised of fix, work--around etcaround etc

The live database may need to be fixedThe live database may need to be fixed

Deploying the new build may take hoursDeploying the new build may take hours

Loss of businessLoss of business


AdditionallyAdditionally

F l f d i di l i h h d


113/414


Faults found in coding or later require changes to the codeFaults found in coding or later require changes to the code

librarylibrary

Documentation and help files will have to be changedDocumentation and help files will have to be changed

Every fault requires investigating and loggingEvery fault requires investigating and logging

Administration costsAdministration costs


The cost of not TestingThe cost of not Testing

Wh i h f i ?Wh t i th t f t ti ?


114/414


What is the cost of testing?What is the cost of testing?

Which is cheaper?Which is cheaper?

Testing and finding faults before releaseTesting and finding faults before release

Not testing and finding faults once the system is liveNot testing and finding faults once the system is live

Companies do not typically have figures for eitherCompanies do not typically have figures for either


SummarySummary

Th f t ti i t fi d f ltTh f t ti i t fi d f lt


115/414


The purpose of testing is to find faultsThe purpose of testing is to find faults

The earlier a fault is found the cheaper (and easier) it is to fThe earlier a fault is found the cheaper (and easier) it is to fixix

Starting testing early will help find faults quickerStarting testing early will help find faults quicker


116/414



High Level Test PlanningHigh Level Test Planning

High Level Test Planning


Look at how a test plan is put togetherLook at how a test plan is put together


117/414


Look at how a test plan is put togetherLook at how a test plan is put together

Understand how it should be used and maintainedUnderstand how it should be used and maintained

Understand why they are so important to a testing projectUnderstand why they are so important to a testing project

High Level Test Planning

What is a test plan?What is a test plan?

A project plan for testingA project plan for testing


118/414


A project plan for testingA project plan for testing

Covering all aspects of testingCovering all aspects of testing


119/414

The Different Sections of a Test Plan

Test plan identifierTest plan identifier

IntroductionIntroduction

T t itT t it

Test deliverablesTest deliverables

Testing tasksTesting tasks

EnvironmentEnvironment


120/414


Test itemsTest items

Features to be testedFeatures to be tested

Features not to be testedFeatures not to be tested

ApproachApproach

Item pass / fail criteriaItem pass / fail criteria

Suspension criteria &Suspension criteria &

resumption criteriaresumption criteria

EnvironmentEnvironment

ResponsibilitiesResponsibilities

Staffing and training needsStaffing and training needs

SchedulesSchedules

Risks and contingenciesRisks and contingencies ApprovalsApprovals

Based upon IEEE 829Based upon IEEE 829--1998 standard for1998 standard forsoftware test documentationsoftware test documentation


121/414


122/414

Test Items

The various items to be used in the testsThe various items to be used in the tests

The software itemsThe software items


123/414


The software itemsThe software items

Their versions numbers / identifiersTheir versions numbers / identifiers

How they will be handed over to testingHow they will be handed over to testing

References to relevant documentsReferences to relevant documents

Features to Be Tested

List all features of the SUT that will be tested under thisList all features of the SUT that will be tested under this

planplan


124/414


Features Not to Be Tested

List all features of the SUT that will not be tested underList all features of the SUT that will not be tested under

this planthis plan


125/414


Between the test items, features to be tested and features not tBetween the test items, features to be tested and features not t

o beo be

tested we have scope of the projecttested we have scope of the project


126/414


127/414

Suspension criteria & resumption requirements

Reasons that would cause testing to be suspendedReasons that would cause testing to be suspended

Steps necessary to resume testingSteps necessary to resume testing


128/414


p y gp y g

Test Deliverables

Everything that goes to make up the testsEverything that goes to make up the tests

All documentationAll documentation


129/414


e.g. Specification, test plans, procedures, reportse.g. Specification, test plans, procedures, reports

Code releasesCode releases

Testing toolsTesting tools

Test management tools, automation tools, excel, word etcTest management tools, automation tools, excel, word etc

Test systemsTest systems

Manual and automated test casesManual and automated test cases

Testing Tasks

Preparation to perform testingPreparation to perform testing

Test case identificationTest case identification Test case designTest case design


130/414


Test data storageTest data storage

Baseline applicationBaseline application

Special skills neededSpecial skills needed

Spreadsheet skills, test analysis, automation etcSpreadsheet skills, test analysis, automation etc

InterInter--dependenciesdependencies

Environment

Requirements for test environmentRequirements for test environment

Hardware & softwareHardware & software


131/414


PCs, servers, routers etcPCs, servers, routers etc

SUT, interfaced applications, databasesSUT, interfaced applications, databases

ConfigurationConfiguration

Maybe operating systems or middleware to test againstMaybe operating systems or middleware to test against

FacilitiesFacilities

Office space, desks, internet accessOffice space, desks, internet access


132/414

Staffing and Training Needs

Staff requiredStaff required

Test managers, team leaders, testers, test analystsTest managers, team leaders, testers, test analysts


133/414


Skill levels requiredSkill levels required

Automation experienceAutomation experience

Spreadsheet skills, etcSpreadsheet skills, etc

Training requirementsTraining requirements

Tools specific trainingTools specific training

Refresher courses, etc.Refresher courses, etc.

Overall project resource requirementsOverall project resource requirements

Schedule

Timescales, dates and milestonesTimescales, dates and milestones

Resources required to meet milestonesResources required to meet milestones


134/414


Availability of software and environmentAvailability of software and environment

DeliverablesDeliverables

Risks and Contingencies

What might go wrong?What might go wrong?

Actions for minimising impact on testing should things goActions for minimising impact on testing should things go


135/414


wrongwrong

Approvals

Who has approved the test planWho has approved the test plan

Names and dates of approvalNames and dates of approval


136/414


Why is it so importantWhy is it so important

Evidence that the document has been viewedEvidence that the document has been viewed

Shows that the approach has been agreed and has the backingShows that the approach has been agreed and has the backing

of those who matterof those who matter

You have commitment, now make them stick to it!You have commitment, now make them stick to it!


137/414



138/414



Acceptance TestingAcceptance Testing

Acceptance Testing


Understand what acceptance testing isUnderstand what acceptance testing is

Wh ld d i


139/414


Why you would want to do itWhy you would want to do it

How you would plan it and prepare for itHow you would plan it and prepare for it

What you need to actually do itWhat you need to actually do it

Understand the different types of acceptance testingUnderstand the different types of acceptance testing

Introduction

What is Acceptance Testing ?What is Acceptance Testing ?

"Formal testing conducted to enable a user, customer or other"Formal testing conducted to enable a user, customer or other

authorised entity to determine whether to accept a system orauthorised entity to determine whether to accept a system or


140/414


authorised entity to determine whether to accept a system orauthorised entity to determine whether to accept a system or

component."component."

User Acceptance Testing

What is User Acceptance Testing (UAT)?What is User Acceptance Testing (UAT)?

Exactly what it says it is!!Exactly what it says it is!!


141/414


Users of end product conducting the testsUsers of end product conducting the tests

The setThe set--up will represent a working Environmentup will represent a working Environment

Covers all areas of a project, not just the systemCovers all areas of a project, not just the system

A.K.A Business Acceptance Testing or Business ProcessA.K.A Business Acceptance Testing or Business Process

TestingTesting

UnitTest

Where does it fit in?


142/414


SystemsTest

UAT

Alpha /

Beta

Service LevelTesting

Operational User Acceptance

BusinessRequirements

A typical development life-cycle V-Model


143/414


level

time

Operational

Requirements

DesignSpecifications

FunctionalSpecification

ComponentTesting

SystemTesting

IntegrationTesting (large)

User Acceptance

Testing

ProgramSpecification


144/414

Why Plan?

If you donIf you dont then how do you know you have achievedt then how do you know you have achieved

what you set out to do!what you set out to do!

Avoids repetitionAvoids repetition


145/414


Avoids repetitionAvoids repetition

Test according to code releasesTest according to code releases

Makes efficient and effective use of time and resourcesMakes efficient and effective use of time and resources

Things to consider

TimescalesTimescales

ResourcesResources


146/414


AvailabilityAvailability

Preparing your tests

Take a logical approachTake a logical approach

Identify the business processesIdentify the business processes


147/414


Separate into everyday business scenariosSeparate into everyday business scenarios

How the business fits together...

The Business


148/414


Process Process Process

Task Task Task Task Task Task Task TaskTask


149/414

Running the Tests

Order of TestsOrder of Tests

Confidence ChecksConfidence Checks


150/414


Automated and Manual test runsAutomated and Manual test runs

Contract Acceptance

A demonstration of the acceptance criteriaA demonstration of the acceptance criteria

Acceptance criteria will have been defined in the contractAcceptance criteria will have been defined in the contract


151/414


Before the software is accepted it is necessary to show thatBefore the software is accepted it is necessary to show thatits matches itsits matches its specification as defined in the criteriaspecification as defined in the criteria

Alpha Testing

Letting your customers do your testingLetting your customers do your testing

Requires a stable version of the softwareRequires a stable version of the software


152/414


People who represent your target market use the product inPeople who represent your target market use the product inthe same way(s) as if they had bought the finished versionthe same way(s) as if they had bought the finished version

Alpha testing is conducted inAlpha testing is conducted in--house (at the developers site)house (at the developers site)

Beta Testing

As Alpha testing but users perform tests at their siteAs Alpha testing but users perform tests at their site


153/414


Acceptance Testing

SummarySummary

You can UAT anything!You can UAT anything!


154/414


Every Deliverable should pass through UATEvery Deliverable should pass through UAT

User representation is VITALUser representation is VITAL

If the product does not pass UAT then a decision aboutIf the product does not pass UAT then a decision about

implementation needs to be madeimplementation needs to be made



155/414


Integration Testing in the LargeIntegration Testing in the Large


156/414

Integration Testing in the Large

Integration testingIntegration testing Testing performed to expose faults in the interfaces and in theTesting performed to expose faults in the interfaces and in the

interaction between integrated componentsinteraction between integrated components


157/414


Integration testing in the largeIntegration testing in the large

Integration Testing in the Large is testing performed to exposeIntegration Testing in the Large is testing performed to expose

faults in the interfaces and in the interaction between systemsfaults in the interfaces and in the interaction between systems

Why do we need to do integration testing in the largeWhy do we need to do integration testing in the large

Just because the SUT works in the test lab doesnJust because the SUT works in the test lab doesnt mean it willt mean it willwork in the real worldwork in the real world


158/414

Env

EnvEnv

EnvEnv

Env

SUT

SUT SUT

SUT SUT

SUT

A typical system

Data Model

Data Model


159/414


DatabasesDatabases

DataData Ext apps.Ext apps.


Integration Testing on a Large ScaleIntegration Testing on a Large Scale

Test the way the SUT interfaces with external systemsTest the way the SUT interfaces with external systems


160/414


The elements may run on the same and / or differentThe elements may run on the same and / or differentplatformsplatforms

They may be located in separate locations and useThey may be located in separate locations and usecommunication protocols to pass data back & forthcommunication protocols to pass data back & forth


Typical approachTypical approach

Test the systemTest the system

In a test environmentIn a test environment

Use stubs & drivers where external systems arenUse stubs & drivers where external systems arent availablet available


161/414


Use stubs & drivers where external systems arenUse stubs & drivers where external systems aren t availablet available

Test the systemTest the system in situin situ

In a replica of the production / live environmentIn a replica of the production / live environment

Use stubs & drivers where external systems arenUse stubs & drivers where external systems arent availablet available

Test the system and itsTest the system and its interfaces with other systemsinterfaces with other systems

In a replica of the production / live environmentIn a replica of the production / live environment

Access test versions of the external systemsAccess test versions of the external systems


Fundamentally we are still looking atFundamentally we are still looking at

DATADATA


162/414


& how that data is obtained, processed, manipulated, stored& how that data is obtained, processed, manipulated, stored

and made available or transported for use elsewhere within orand made available or transported for use elsewhere within or

outside theoutside the organisationorganisation


Additional challenges posed by large scale:Additional challenges posed by large scale:

Multiple PlatformsMultiple Platforms

Communications between platformsCommunications between platforms


163/414


Management of the environmentsManagement of the environments

Coordination of changesCoordination of changes

Different technical skills requiredDifferent technical skills required

The approach used


164/414


Integration Testing

Planning testingPlanning testing

We need to determine when and at what levels IntegrationWe need to determine when and at what levels Integration

Testing will be performedTesting will be performed


165/414


Identify the boundariesIdentify the boundaries

A similar process will need to be followed for all elementsA similar process will need to be followed for all elements Once these elements have been tested individually they areOnce these elements have been tested individually they are

further integrated to support the business processfurther integrated to support the business process

Integration Testing

Approaches to integration testingApproaches to integration testing

IncrementalIncremental

Top downTop down


166/414


BottomBottom--upup FunctionalFunctional

NonNon--incrementalincremental

Big bangBig bang


SummarySummary

Integration testing is needed to test how components interactIntegration testing is needed to test how components interact

with each other and how the system under test works withwith each other and how the system under test works with

other systemsother systems


167/414


y

Large scale integration testing is needed to test how theLarge scale integration testing is needed to test how the

system under test works with other systemssystem under test works with other systems

Just because a system works in a test environment doesnJust because a system works in a test environment doesntt

mean it will work in a live environmentmean it will work in a live environment



168/414


NonNon--Functional System TestingFunctional System Testing

Non-Functional System Testing


Understand what NonUnderstand what Non--Functional System Testing isFunctional System Testing is


169/414


Understand the need for nonUnderstand the need for non--functional system testingfunctional system testing

Look at the various types of nonLook at the various types of non--functional system testingfunctional system testing


NonNon--Functional System Testing is defined as:Functional System Testing is defined as:

Testing Of Those requirements that do not relate to theTesting Of Those requirements that do not relate to the

Functionality e.g. Performance, Load, usability, security etcFunctionality e.g. Performance, Load, usability, security etc


170/414


Users, generally, focus on the Functionality of the systemUsers, generally, focus on the Functionality of the system


171/414

Security Testing

Testing whether the system meets specified securityTesting whether the system meets specified security

requirementsrequirements

How easy is it for an unauthorised user to access theHow easy is it for an unauthorised user to access the

system?system?


172/414


How easy is it for an unauthorised person to access the data?How easy is it for an unauthorised person to access the data?

Usability Testing

Testing the ease with which users can learn and use theTesting the ease with which users can learn and use the

productproduct

Employ people to use the application as a user and studyEmploy people to use the application as a user and study

how they use ithow they use it


173/414


A betaA beta--test may be a cheap way of doing Usability testingtest may be a cheap way of doing Usability testing

There is no simple way to examine HOW people use theThere is no simple way to examine HOW people use the

productproduct

Storage Testing

Testing whether the system meets its specified storageTesting whether the system meets its specified storage

objectivesobjectives

Study how memory and storage is used by the productStudy how memory and storage is used by the product


174/414


Predict when extra capacity may be neededPredict when extra capacity may be needed

Installability Testing

Testing concerned with the installation procedures for theTesting concerned with the installation procedures for the

systemsystem

Does the installation work?Does the installation work?


175/414


Is it easy to carry out?Is it easy to carry out?

Does it uninstall correctly?Does it uninstall correctly?

Documentation Testing

Testing concerned with the accuracy of the documentationTesting concerned with the accuracy of the documentation


176/414


Recovery Testing

Testing aimed at verifying the systemTesting aimed at verifying the systems ability to recover froms ability to recover from

varying degrees of failurevarying degrees of failure

Should include recovery from system backShould include recovery from system back--upsups


177/414



178/414

Stress Testing

Assesses individual components by exercising them to andAssesses individual components by exercising them to and

beyond the limits of expected usebeyond the limits of expected use


179/414


Performance Testing

Tests the efficiency of individual components of an applicationTests the efficiency of individual components of an application


180/414


Volume Testing

Testing where the system is subjected to large volumes ofTesting where the system is subjected to large volumes of

datadata


181/414



SummarySummary

Just because a systemJust because a systems functions have been tested doesns functions have been tested doesntt

mean that testing is completemean that testing is complete


182/414


There are a range of nonThere are a range of non--functional tests that need to befunctional tests that need to be

performed upon a systemperformed upon a system



183/414


Functional System TestingFunctional System Testing


184/414

Functional System Testing

What is System Testing?What is System Testing?

Testing of the complete systemTesting of the complete system

May be the last step in integration testing in the smallMay be the last step in integration testing in the small

May be the first time that enough of the system has been putMay be the first time that enough of the system has been puttogether to make a working systemtogether to make a working system


185/414


y g y py g y ptogether to make a working systemtogether to make a working system

Ideally done by an independent test teamIdeally done by an independent test team

Two typesTwo types -- functional and nonfunctional and non--functionalfunctional


A Functional Requirement isA Functional Requirement is

A requirement that specifies a function that a system or systemA requirement that specifies a function that a system or system

component must performcomponent must perform


186/414


Functional System testing is geared to checking the functionFunctional System testing is geared to checking the function

of the system against specificationof the system against specification

May be requirements based or business process basedMay be requirements based or business process based


Testing Based on RequirementsTesting Based on Requirements

Requirements specification used to derive test casesRequirements specification used to derive test cases

System is tested to ensure the requirements can be metSystem is tested to ensure the requirements can be met


187/414


y qy q


Testing carried out against Business processesTesting carried out against Business processes

Based on expected use of the systemBased on expected use of the system

Builds use casesBuilds use cases -- test cases that reflect actual or expectedtest cases that reflect actual or expecteduse of the systemuse of the system


188/414


ppuse of the systemuse of the system


189/414



190/414



191/414


Integration Testing in the SmallIntegration Testing in the Small

Integration Testing in the Small


Understand whatUnderstand what Integration Testing in the SmallIntegration Testing in the Small isis

Look at how & why we doLook at how & why we do Integration Testing in the SmallIntegration Testing in the Small


192/414



Integration testingIntegration testing

Testing performed to expose faults in the interfaces and in theTesting performed to expose faults in the interfaces and in the

interaction between integrated componentsinteraction between integrated components

Integration testing in the smallIntegration testing in the small


193/414


g g

Testing performed to expose faults in the interfaces and in theTesting performed to expose faults in the interfaces and in the

interaction between componentsinteraction between components

A sample system

DatabasesDatabases


194/414


Externalsystems

Externalsystems

The SUT

- made up of 000s of units

The SUT

- made up of 000s of units


All components of a computer application work on onlyAll components of a computer application work on only

one thingone thing

DATADATA

Data is passed from one component to anotherData is passed from one component to another


195/414


Data is passed from one component to anotherData is passed from one component to another

Data is passed from one system to anotherData is passed from one system to another

Software components will be required to add, amend, &Software components will be required to add, amend, &

delete information, validate the information and report on thedelete information, validate the information and report on theinformationinformation


Integration testingIntegration testing

Testing should start with the smallest, definable componentTesting should start with the smallest, definable component

Building tested components up to the next level of definitionBuilding tested components up to the next level of definition


196/414


Eventually the complete business processes is testedEventually the complete business processes is tested End toEnd to

EndEnd


197/414


Once components have been tested we can link themOnce components have been tested we can link them

together to see if they can work togethertogether to see if they can work together

This is Integration Testing in the SmallThis is Integration Testing in the Small

Linking components together to ensure that they communicateLinking components together to ensure that they communicatecorrectlycorrectly


198/414


Gradually linking more and more components togetherGradually linking more and more components together

Prove that the data is passed between components as requiredProve that the data is passed between components as required

Steadily increase the number of components and create & testSteadily increase the number of components and create & test

subsub--systems and then the complete systemsystems and then the complete system

The approach used


199/414


Integration Testing

Planning testingPlanning testing

We need to determine when and at what levels IntegrationWe need to determine when and at what levels Integration

Testing will be performedTesting will be performed

Identify the boundariesIdentify the boundaries


200/414


A similar process will need to be followed for all elementsA similar process will need to be followed for all elements Once these elements have been tested individually they areOnce these elements have been tested individually they are

further integrated to support the business processfurther integrated to support the business process

Integration Testing

Stubs and driversStubs and drivers

Used to emulate theUsed to emulate the missing bitsmissing bits

External systemsExternal systems

SubSub--systemssystems Missing componentsMissing components


201/414


May need to be written specifically for the SUTMay need to be written specifically for the SUT

Integration Testing

Approaches to integration testingApproaches to integration testing

IncrementalIncremental

Top downTop down

BottomBottom--upup FunctionalFunctional


202/414


NonNon--incrementalincremental Big bangBig bang


SummarySummary

Integration testing is needed to test how components interactIntegration testing is needed to test how components interact

with each other and how the system under test works withwith each other and how the system under test works with

other systemsother systems

I i i i h ll i d i h hI t ti t ti i th ll i d ith th


203/414


Integration testing in the small is concerned with theIntegration testing in the small is concerned with the

interaction between the various components of a systeminteraction between the various components of a system



204/414


Component TestingComponent Testing

Component Testing


Understand what Component Testing isUnderstand what Component Testing is

Look at the Component Testing ProcessLook at the Component Testing Process


205/414


Look at the myriad types of Component TestingLook at the myriad types of Component Testing

Component Testing

What is component testing?What is component testing?

The testing of an individual software componentThe testing of an individual software component

What is a component?What is a component?


206/414


A minimal software item for which a separate specification isA minimal software item for which a separate specification is

availableavailable

Component Testing

Component testing is the lowest form of testingComponent testing is the lowest form of testing

At the bottom of the VAt the bottom of the V--modelmodel

Each component is tested in isolationEach component is tested in isolation Prior to being integrated into the systemPrior to being integrated into the system


207/414


Involves testing the code itselfInvolves testing the code itself Test the code in the gr

iseb software testing presentation

Documents