Software Testing Using Software Testing Using Model ProgramModel Program
DESIGN BYDESIGN BY
HONG NGUYEN & SHAH RAZAHONG NGUYEN & SHAH RAZA
Dec 05, 2005Dec 05, 2005
OverviewOverview Software Testing Using Model ProgramSoftware Testing Using Model Program
Briefly introduction of M-mp testingBriefly introduction of M-mp testing What is M-mp testing?What is M-mp testing? How it work?How it work? Why we need program base testingWhy we need program base testing
The problem of software testingThe problem of software testing Oracle testOracle test
N-Version programmingN-Version programming What is N-Version Programming?What is N-Version Programming? DisadvantagesDisadvantages AdvantagesAdvantages
Overview (continues)Overview (continues)
M-mp testingM-mp testing Framework for M-mp testingFramework for M-mp testing Asymptotic behavior of M-mp testingAsymptotic behavior of M-mp testing Input domain partitioningInput domain partitioning An M-mp ExperimentAn M-mp Experiment
Background and motivationBackground and motivation Selecting the applicationSelecting the application Phases of the experimentPhases of the experiment
ConclusionConclusion
IntroductionIntroduction What is M-mp testing?What is M-mp testing?
Testing using M Model programTesting using M Model program Abbreviated to ‘M-mp testing’Abbreviated to ‘M-mp testing’ It is alternative to software testing base on It is alternative to software testing base on
manual outcome predictionmanual outcome prediction M-mp testing is base on N-Version programming M-mp testing is base on N-Version programming
technique technique Which is apply specially in the well-known software Which is apply specially in the well-known software
fault tolerant technique.fault tolerant technique.
Introduction (Continues)Introduction (Continues)
How it workHow it work Model program implements selected part of functional Model program implements selected part of functional
specification of the software to be tested.specification of the software to be tested. The M-mp testing strategy requires that M (M>= 1) The M-mp testing strategy requires that M (M>= 1)
Model programs as well as the program under test, should be Model programs as well as the program under test, should be independently developedindependently developed
P and M model program are then subjected to the same test P and M model program are then subjected to the same test data. data.
Difference analysis is conducted on the outputs and Difference analysis is conducted on the outputs and appropriate corrective action takenappropriate corrective action taken
P and the M model program jointly constituted an P and the M model program jointly constituted an approximate test oracle (out come correct or not)approximate test oracle (out come correct or not)
Introduction (Continues)Introduction (Continues)
Why we need program base testingWhy we need program base testing For small number of test data, it may be For small number of test data, it may be
feasible to work out the expected output feasible to work out the expected output manually manually
For thorough testing which required large For thorough testing which required large test datasets manual testing could be time test datasets manual testing could be time consuming and error proneconsuming and error prone That is why M-mp base testing may be more That is why M-mp base testing may be more
feasible for those kinds of data setfeasible for those kinds of data set
The problem of software testingThe problem of software testing
The difficult task in software testing is to assess the The difficult task in software testing is to assess the correctness to the outcome of a program that is correctness to the outcome of a program that is subjected to particular test inputsubjected to particular test input
In testing theory, the mechanism to adjudicate on In testing theory, the mechanism to adjudicate on whether or not an output is correct, somewhat whether or not an output is correct, somewhat neglected this problem is referred to as “the oracle neglected this problem is referred to as “the oracle problem.problem.
Testing theory concern with the choice of test and Testing theory concern with the choice of test and testing methods, usually ignore the oracle problems.testing methods, usually ignore the oracle problems.
The problem of software testing The problem of software testing (continues)(continues)
Oracle TestOracle Test It means of determining whether a program passed It means of determining whether a program passed
or failed a test.or failed a test.
The problem of software testing The problem of software testing (continues)(continues)
Oracle test (continues)Oracle test (continues) P is the program to be tested P is the program to be tested Test strategy determines the set of test data to be usedTest strategy determines the set of test data to be used To get the test result we compare P’s output with the oracle To get the test result we compare P’s output with the oracle
‘s output‘s output For small out put oracle manually base oracle technique is feasibleFor small out put oracle manually base oracle technique is feasible But for large industry base application manually base oracle testing But for large industry base application manually base oracle testing
is not feasibleis not feasible One of alternative of manually base oracle is N- Version One of alternative of manually base oracle is N- Version
programmingprogramming
N-Version ProgrammingN-Version Programming
N-Version is comprises N independently written N-Version is comprises N independently written version of the softwareversion of the software
All has the same functional specificationAll has the same functional specification At runtime voting base on majority agreement is used At runtime voting base on majority agreement is used
to decided the probable outputto decided the probable output DisadvantageDisadvantage::
Concern in N-Version systems is the fact that correlated Concern in N-Version systems is the fact that correlated failures may limit the practical gain in reliabilityfailures may limit the practical gain in reliability
N-Version systems should not be casually assumed to be N-Version systems should not be casually assumed to be completed reliable.completed reliable.
N-Version (continues)N-Version (continues)
Disadvantage (continues)Disadvantage (continues) Is the increased development cost because at least three Is the increased development cost because at least three
version being required for a voting system to workversion being required for a voting system to work AdvantageAdvantage
N-Version design appear to offer more reliability that N-Version design appear to offer more reliability that we can gain from any other waywe can gain from any other way
Even though the cost of failure is high:Even though the cost of failure is high: It would be more cost effective to build N-Version systems It would be more cost effective to build N-Version systems
rather than focusing on building ‘one good version’rather than focusing on building ‘one good version’
M-mp Testing (continues)M-mp Testing (continues)
Framework (continues)Framework (continues) In M-mp testing P is primary program (the program under In M-mp testing P is primary program (the program under
test) test) mp1 –mpmp1 –mpMM are M so called ‘model program of P are M so called ‘model program of P M-mp testing treat P and mp1-mpM-mp testing treat P and mp1-mpMM to be same test input to be same test input Any disagreement on the output indicates the presence of Any disagreement on the output indicates the presence of
specification defects Or software faults in at least one of specification defects Or software faults in at least one of programprogram
Once defects are detected and removed. The program are Once defects are detected and removed. The program are re-runre-run
M-mp Testing (continues)M-mp Testing (continues) Framework (continues)Framework (continues)
The cycle is repeat until all disagreements for a particular The cycle is repeat until all disagreements for a particular
dataset are resolveddataset are resolved.. Asymptotic behavior of M-mp testingAsymptotic behavior of M-mp testing
Difference between the M-mp and manual approaches to Difference between the M-mp and manual approaches to testing is the cost of outcome verificationtesting is the cost of outcome verification
M-mp Testing (continues)M-mp Testing (continues) Asymptotic behavior of M-mp testing (continues)Asymptotic behavior of M-mp testing (continues)
On the graph: the cost of M-mp outcome verification is On the graph: the cost of M-mp outcome verification is high initially but it increase only slightly.high initially but it increase only slightly.
The cost growth is modeled as linearThe cost growth is modeled as linear In manual approach, the growth of the output verification In manual approach, the growth of the output verification
cost may be expected to be much steeper.cost may be expected to be much steeper. Input domain partitioningInput domain partitioning
Divided into a standard domain and an exception domainDivided into a standard domain and an exception domain Exception domain may be split into an incomplete data domain and Exception domain may be split into an incomplete data domain and
an invalid data domain.an invalid data domain. The standard domain consists of inputs that a program can process The standard domain consists of inputs that a program can process
‘as is’‘as is’
M-mp Testing (continues)M-mp Testing (continues)
Input domain partitioning (continues)Input domain partitioning (continues) The invalid domain contains the input that a The invalid domain contains the input that a
program can not process it should be rejected with program can not process it should be rejected with appropriate error messageappropriate error message
The incomplete data domain is ‘between’ the The incomplete data domain is ‘between’ the standard and invalid data domainsstandard and invalid data domains
It is made up of inputs that the program has to It is made up of inputs that the program has to completed with default values before processing themcompleted with default values before processing them
An M-mp ExperimentAn M-mp Experiment Background and motivationBackground and motivation
M-mp Testing (continues)M-mp Testing (continues) Background and motivation (continues)Background and motivation (continues)
The primary program is a scheduling application, a non-The primary program is a scheduling application, a non-interactive data processing subsystem of a large interactive data processing subsystem of a large logistics management systemlogistics management system
The program can be characterized as algorithmically The program can be characterized as algorithmically complexcomplex
In practice, the testing process in the company was well In practice, the testing process in the company was well defined and supported by standards and procedures but defined and supported by standards and procedures but had several drawbackhad several drawback
Designing test cases based on manual outcome Designing test cases based on manual outcome prediction was not only difficult, error prone and time prediction was not only difficult, error prone and time consumingconsuming
But consequently also unattractive and demoralizingBut consequently also unattractive and demoralizing
M-mp Testing (continues)M-mp Testing (continues) Selecting the applicationSelecting the application
The scheduling application was chosen as the primary program for The scheduling application was chosen as the primary program for following reasonsfollowing reasons
Developer was not familiar with the scheduling domain and applicationDeveloper was not familiar with the scheduling domain and application The cost of developing from scratch could be assesses more The cost of developing from scratch could be assesses more
authenticallyauthentically The risk of correlated failures caused by common design error was The risk of correlated failures caused by common design error was
reducesreduces scheduling application is representative of the family of operation scheduling application is representative of the family of operation
management programmanagement program The time the experiment started, the scheduling subsystem had been in The time the experiment started, the scheduling subsystem had been in
use for more than yearuse for more than year This seem a good opportunity to test the fault detection ability of M-mp This seem a good opportunity to test the fault detection ability of M-mp
testing, as the likelihood of finding defect was expected to be lowtesting, as the likelihood of finding defect was expected to be low
M-mp Testing (continues)M-mp Testing (continues)
Phase of the experimentPhase of the experiment The experiment entailed a test design, a test execution and The experiment entailed a test design, a test execution and
test evaluation phase as depictedtest evaluation phase as depicted Test design phase comprised test data selectionTest design phase comprised test data selection
Which includes activities related to test data generation, Model Which includes activities related to test data generation, Model program design and model program implantationprogram design and model program implantation
The test execution phase includes activities relating to The test execution phase includes activities relating to setting up the test environment and executing the programsetting up the test environment and executing the program
The test evaluation phase includes the disagreement The test evaluation phase includes the disagreement analysis procedures whereby disagreements between the analysis procedures whereby disagreements between the program need to be arbitrated.program need to be arbitrated.
ConclusionConclusion
M-mp approach could test a scheduling M-mp approach could test a scheduling program more adequately than manually program more adequately than manually designed test and at lower costdesigned test and at lower cost
M-mp testing should be judged on a case by M-mp testing should be judged on a case by case basiccase basic
M-mp approach will tend to be better option M-mp approach will tend to be better option than the manual one when the number of tests than the manual one when the number of tests required to achieve a particular adequacy level required to achieve a particular adequacy level become largebecome large