software engineering ch-13

3/18/2009 S.Sreenivasa Rao 1

Chapter 13Chapter 13Software Testing StrategiesSoftware Testing Strategies


Software TestingSoftware Testing

Testing is the process of exercising aTesting is the process of exercising aprogram with the specific intent of findingprogram with the specific intent of findingerrors prior to delivery to the end user.errors prior to delivery to the end user.

3/18/2009 S.Sreenivasa Rao

Test strategy provides a road map;Test strategy provides a road map;Steps to be conductedSteps to be conductedWhenWhenHow much effort, time, resourcesHow much effort, time, resources

Test strategyTest strategyTest planningTest planningTest case designTest case designTest executionTest executionResultant data collection and evaluationResultant data collection and evaluation

3


Verification and ValidationVerification and ValidationV&VV&V

Testing is part of V&VTesting is part of V&V

Verification: Are we building the product right?Verification: Are we building the product right?Software correctly implements a specific Software correctly implements a specific functionfunction

Validation: Are we building the right product?Validation: Are we building the right product?Software is traceable to customer Software is traceable to customer requirementsrequirements 4


What Testing ShowsWhat Testing Showserrorserrors

requirements conformancerequirements conformance

performanceperformance

an indicationan indicationof qualityof quality


Who Tests the Software?Who Tests the Software?

developerdeveloper independent testerindependent tester

Understands the system Understands the system

but, will test "gently"but, will test "gently"and, is driven by "delivery"and, is driven by "delivery"

Must learn about the system,Must learn about the system,but, will attempt to break itbut, will attempt to break itand, is driven by qualityand, is driven by quality


Testing StrategyTesting Strategyunit testunit test integrationintegration

testtest

validationvalidationtesttest

systemsystemtesttest


Levels of Testing

Client Needs

Requirements

Design

Code

Acceptance Testing

System Testing

Integration Testing

Unit testing


Requirement Requirement --> high> high--order tests: order tests: validation test, System testvalidation test, System testDesign Design --> Integration test> Integration testCode Code --> unit test> unit test

9


Testing StrategyTesting Strategy

We begin by We begin by ‘‘testingtesting--inin--thethe--smallsmall’’ and move toward and move toward ‘‘testingtesting--inin--thethe--largelarge’’For conventional softwareFor conventional software

The module (component) is our initial focusThe module (component) is our initial focusIntegration of modules followsIntegration of modules follows

For OO softwareFor OO softwareour focus when our focus when ““testing in the smalltesting in the small”” changes from an changes from an individual module (the conventional view) to an OO class individual module (the conventional view) to an OO class

that encompasses attributes and operations and implies that encompasses attributes and operations and implies communication and collaborationcommunication and collaboration


Strategic IssuesStrategic IssuesSpecify product requirements in a quantifiable manner long befoSpecify product requirements in a quantifiable manner long before re testing.testing.State testing objectives explicitly. State testing objectives explicitly. Understand the users of the software and develop a profile for eUnderstand the users of the software and develop a profile for each ach user category.user category.Develop a testing plan that emphasizes Develop a testing plan that emphasizes ““rapid cycle testing.rapid cycle testing.””Build Build ““robustrobust”” software that is designed to test itselfsoftware that is designed to test itselfUse effective formal technical reviews as a filter prior to testUse effective formal technical reviews as a filter prior to testingingConduct formal technical reviews to assess the test strategy andConduct formal technical reviews to assess the test strategy and test test cases themselves. cases themselves. Develop a continuous improvement approach for the testing Develop a continuous improvement approach for the testing process. process.


Unit TestingUnit Testing

modulemoduleto beto be

testedtested

test casestest cases

resultsresults

softwaresoftwareengineerengineer


Unit TestingUnit Testing

interface interface local data structureslocal data structuresboundary conditionsboundary conditionsindependent pathsindependent pathserror handling pathserror handling paths

modulemoduleto beto be

testedtested



Unit Test EnvironmentUnit Test Environment

ModuleModule

stubstub stubstub

driverdriver

interface interface

local data structureslocal data structures

boundary conditionsboundary conditions

independent pathsindependent paths

error handling pathserror handling paths

RESULTSRESULTS



Integration Testing StrategiesIntegration Testing Strategies

Options:Options:•• the the ““big bangbig bang”” approachapproach•• an incremental construction strategyan incremental construction strategy


Top Down IntegrationTop Down Integration

top module is tested with top module is tested with stubsstubs

stubs are replaced one at stubs are replaced one at a time, "depth first"a time, "depth first"

as new modules are integrated, as new modules are integrated, some subset of tests is resome subset of tests is re--runrun

AA

BB

CC

DD EE

FF GG


BottomBottom--Up IntegrationUp Integration

drivers are replaced one at a drivers are replaced one at a time, "depth first"time, "depth first"

worker modules are grouped into worker modules are grouped into builds and integratedbuilds and integrated

AA

BB

CC

DD EE

FF GG

clustercluster


Sandwich TestingSandwich Testing

Top modules areTop modules aretested with stubstested with stubs

Worker modules are grouped into Worker modules are grouped into builds and integratedbuilds and integrated

AA

BB

CC

DD EE

FF GG

clustercluster


Regression testingRegression testing

to ensure that changes do not introduce to ensure that changes do not introduce unintended behavior or additional errors.unintended behavior or additional errors.

In context of an integration test: reIn context of an integration test: re--execution of some subset of testsexecution of some subset of testsAny correction Any correction --> change of software> change of software

19


Smoke TestingSmoke TestingA common approach for creating A common approach for creating ““daily buildsdaily builds”” for product softwarefor product softwareSmoke testing steps:Smoke testing steps:

Software components that have been translated into code are inteSoftware components that have been translated into code are integrated grated into a into a ““build.build.””

A build includes all data files, libraries, reusable modules, anA build includes all data files, libraries, reusable modules, and d engineered components that are required to implement one or moreengineered components that are required to implement one or moreproduct functions.product functions.

A series of tests is designed to expose errors that will keep thA series of tests is designed to expose errors that will keep the build e build from properly performing its function. from properly performing its function.

The intent should be to uncover The intent should be to uncover ““show stoppershow stopper”” errors that have the errors that have the highest likelihood of throwing the software project behind schedhighest likelihood of throwing the software project behind schedule.ule.

The build is integrated with other builds and the entire productThe build is integrated with other builds and the entire product (in its (in its current form) is smoke tested daily. current form) is smoke tested daily.

The integration approach may be top down or bottomThe integration approach may be top down or bottom up.up.


Identify critical modulesIdentify critical modules

Address several software requirementsAddress several software requirementsHas a high level of control (high in the program Has a high level of control (high in the program structure)structure)Is complex or error proneIs complex or error proneHas definite performance requirementsHas definite performance requirements

21


ObjectObject--Oriented TestingOriented Testing

testing strategy changestesting strategy changesthe concept of the the concept of the ‘‘unitunit’’ broadens due to encapsulationbroadens due to encapsulationintegration focuses on classes and their executionintegration focuses on classes and their executionvalidation uses conventional black box methodsvalidation uses conventional black box methods

test case design draws on conventional methods, but test case design draws on conventional methods, but also encompasses special featuresalso encompasses special features


OOT StrategyOOT Strategy

class testing is the equivalent of unit testingclass testing is the equivalent of unit testingoperations within the class are testedoperations within the class are testedthe state behavior of the class is examinedthe state behavior of the class is examined

integration applied three different strategiesintegration applied three different strategiesthreadthread--based testingbased testing——integrates the set of classes integrates the set of classes required to respond to one input or eventrequired to respond to one input or eventuseuse--based testingbased testing——integrates the set of classes required integrates the set of classes required to respond to one use caseto respond to one use casecluster testingcluster testing——integrates the set of classes required to integrates the set of classes required to demonstrate one collaborationdemonstrate one collaboration


Validation TestingValidation Testing

Validation succeeds when software functions in a Validation succeeds when software functions in a manner that can be reasonably expected by the manner that can be reasonably expected by the customer.customer.Reasonable expectation are defined in the Software Reasonable expectation are defined in the Software Requirement SpecificationRequirement Specification

FunctionalFunctionalBehavioralBehavioralPerformancePerformanceDocumentationDocumentationUsability Usability ……. .

Alpha/Beta testingAlpha/Beta testingFocus is on customer usageFocus is on customer usage

24


High Order Testing: High Order Testing: System testingSystem testing

System testingSystem testingFocus is on system integrationFocus is on system integrationRecovery testingRecovery testing

forces the software to fail in a variety of ways and forces the software to fail in a variety of ways and verifies that recovery is properly performedverifies that recovery is properly performedAutomatic: reAutomatic: re--initialization, check pointing, data initialization, check pointing, data recovery, restartrecovery, restartBy human intervention ; meanBy human intervention ; mean--timetime--toto--repairrepair

Security testingSecurity testingverifies that protection mechanisms built into a system verifies that protection mechanisms built into a system will, in fact, protect it from improper penetrationwill, in fact, protect it from improper penetration

Stress testingStress testingexecutes a system in a manner that demands resources executes a system in a manner that demands resources in abnormal quantity, frequency, or volumein abnormal quantity, frequency, or volume

Performance TestingPerformance Testingtest the runtest the run--time performance of software within the time performance of software within the context of an integrated systemcontext of an integrated system


Debugging: Debugging: A Diagnostic ProcessA Diagnostic Process


The Debugging ProcessThe Debugging Processtest casestest cases

resultsresults

DebuggingDebugging

suspectedsuspectedcausescauses

identifiedidentifiedcausescauses

correctionscorrections

regressionregressionteststests

new testnew testcasescases


Debugging EffortDebugging Effort

time requiredtime requiredto diagnose theto diagnose thesymptom andsymptom anddetermine thedetermine thecausecause

time requiredtime requiredto correct the errorto correct the errorand conductand conductregression testsregression tests


Symptoms & CausesSymptoms & Causes

symptomsymptomcausecause

symptom and cause may be symptom and cause may be geographically separated geographically separated

symptom may disappear when symptom may disappear when another problem is fixedanother problem is fixed

cause may be due to a cause may be due to a combination of noncombination of non--errors errors

cause may be due to a system cause may be due to a system or compiler erroror compiler error

cause may be due to cause may be due to assumptions that everyone assumptions that everyone believesbelieves

symptom may be intermittentsymptom may be intermittent


Debugging tacticsDebugging tactics

Brute forceBrute forceDump the memoryDump the memory

BacktrackingBacktrackingStart from presence of error, go backward in the code manuallyStart from presence of error, go backward in the code manually

Cause eliminationCause eliminationCause hypothesis Cause hypothesis ……

30


Consequences of BugsConsequences of Bugs

damagedamage

mildmild annoyingannoying

disturbingdisturbingseriousserious

extremeextremecatastrophiccatastrophic

infectiousinfectious

Bug TypeBug Type

Bug Categories:Bug Categories: functionfunction--related bugs, related bugs, systemsystem--related bugs, data bugs, coding bugs, related bugs, data bugs, coding bugs, design bugs, documentation bugs, standards design bugs, documentation bugs, standards violations, etc.violations, etc.


Debugging: Final ThoughtsDebugging: Final Thoughts

thinkthink about the about the symptom you're seeing.symptom you're seeing.

Use toolsUse tools (e.g., dynamic debugger) to gain (e.g., dynamic debugger) to gain more insight.more insight.

get helpget help from someone else.from someone else.

Be absolutely sure to Be absolutely sure to conduct regression testsconduct regression testswhen you do "fix" the bug.when you do "fix" the bug.

1.1.

2.2.

3.3.

4.4.

software engineering ch-13

Documents