Alexandru Lupu 12 of November, 2010 Software DeveloperI IC RD R SEI SI

Unit Testing Techniques by exampleTerms and classifications are from Standard for Software Component Testing British Computer Society, SIGIST

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGContent OverviewFunctional Design TechniquesEquivalence PartitioningBoundary AnalysisState TransitionCause-Effect GraphingSyntax

IntroductionWhy unit testing?Unit & Module

Structural Design Techniques

StatementBranch/DecisionData FlowBranch ConditionLCSAJ

Other TechniquesRandomAll pairs

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGWhy unit testing?The cost of software bugs is staggering.

The National Institute of Standards and Technology estimated that the total annual cost of software bugs in the U.S. alone was $60 billion. The average-per-bug cost for many large companies is for $7000 for bugs caught during system testing, double for bugs caught in production and less than $1000 for bugs caught during unit testing.

Unit Testing

* / ALEXANDRU LUPU / 12.11. 2009 Continental AG

In this presentation unit* means the lowest level of independently testable software. (in C a unit is a function)

Module means a union of several units tighten together by their functionality in a single functional module. (in C a module is a source file)

*In Standard for Software Component Testing used term is ComponentUnit & Module

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(1)DefinitionConsists in splitting of the input & output values in to several partitions.

All values in a partition can reasonably be expected to be treated by the unit in the same way ( equivalent).

It is sufficient to select one test case out of each partition to check the behavior of the program.

Both valid and invalid values are partitioned in this way.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(2)Let's consider following Requirement for a unit called fever indicator for a small medical thermometer:Input: temperature in Celsius degrees, floating point representation, received from a sensor.Output: a diagnostic from { Hypothermia, Normal, Sub fever, Fever, Error} is displayed on the monitor, it is based on the received input as follows:less or equal than 34, but greater than 0 - Hypothermiagreater than 34, but less than 36 - Normalgreater than or equal to 36, but less than 37.5 Sub fevergreater than or equal to 37.5, but less than 42 Feverless than 0; greater than or equal 42 - ErrorRemarks:Input validation remark: Inputs are passed are floating numbers with one digit for decimal part. For all undefined states an Error message is generated.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(3)Input partitioning

First step is to identify the partitions:The most obvious partitions can be described by :

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(4)Second step is to split also the Valid Partition based on the requirements

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(5)Less obvious invalid input equivalence partitions would include any other inputs that can occur not so far included in a partition:

temperature is alphabetictemperature larger than MAX_FLOATtemperature is NULL...

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(6)

Last step is to Choose one element from each partition and add it to test suite

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Equivalence Partitioning(7)In a more complex case with more than one input and a different mapping of output partitions (different from 1:1) partitioning is a less trivial case. Imagine the case when we have as input also blood pressure and we must establish a more elaborate list of diagnostics.Next step will be to describe output partitions and make sure that all of them are exercised, for this simple example this action is not necessary.Equivalence partitioning provides a warranty that there is a coverage of input and output partitions, but because it's a functional test technique it does not provide any information about code coverage.Metric for Equivalence Partitioning is:

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Boundary Value Analysis (1)Definition Tests are designed to include representatives of boundary values (valid and invalid). A partition's boundaries are normally defined by the values of the boundaries between partitions, however where partitions are disjoint the minimum and maximum values in the range which makes up the partition are used. The values could be either input or output ranges of a software unit. Since these boundaries are common locations for errors that result in software faultsthey are frequently used in test cases.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Boundary Value Analysis (2)As example we will same application as for Equivalence Partitioning.

Boundary values are: {-0.1,0,0.1}; {33.9,34,34.1}; {35.9,36,36.1}; {37.4,37.5,37.6}; {41.9,42,42.1}; {MAX_FLOAT,MIN_FLOAT,MAX_FLOAT+1,MIN_FLOAT-1}

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Boundary Value Analysis (3)Measurement

For previous example test cases: {-0.1; 33.9; 34; 41.9; 42;} provide 5/19*100 =26.31% boundary coverage

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - State Transition Testing(1)DesignThis technique uses a model of the states the unit may have, the transitions between those states, the events which cause those transitions, and the actions which may result from those transitions.model shall comprise states, transitions, events, actions and their relationships. The model will typically be represented as a state transition diagram, state transition model, or a state table.states of the model shall be disjoint, identifiable and finite in number. events cause transitions between states, and transitions can return to the same state where they began. Events will be caused by inputs to the unit, and actions in the state transition model may cause outputs from the unit.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - State Transition(2) Because a thermometer has a linear behavior, and given problem contains as output an enumeration of states we can use the same example for creating a state model.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - State Transition(3)MetricsCoverage items are sequences of one or more transitions between states on the model.

Coverage calculation: For n transitions, the coverage measure is the percentage of all valid sequences of n transitions exercised during test. This is known as (n - 1) switch coverage. (for a single transition term is 0-switch coverage).

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - State Transition(4)Test cases are designed to exercise valid transitions between states. Test cases may also be designed to test that unspecified transitions cannot be induced. 0-switch coverage

TestCaseStartStateInputOutputFinishState1S1T = 35NormalS22S1T = -1ErrorS53S2T = 36SubFeverS34S3T = 37.5FeverS45S4T= 43ErrorS56S4T= 36SubFeverS37S5T = 34HypotermiaS18S5T= 41FeverS4

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - State Transition(5)1-switch coverage

TestCaseStartStateInputNext StateOutputFinishState1S1T = 36S2SubFeverS32S2T = 40S3FeverS43S2T = -1S1ErrorS54S3 T= 43S4ErrorS55S3 T= 1S2HypotermiaS16S4T = 35S3NormalS27S5T = 35S1NormalS28S5T=37.5S4FeverS3..................

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Cause-Effect Graphing(1)DesignThis technique uses a model of the logical relationships between causes and effects for the unit under test. Each cause is expressed as a condition, which is either true of false (i.e. a Boolean) on an input, or combination of inputs. Each effect is expressed as a Boolean expression representing an outcome, or a combination of outcomes.The model is typically represented as a Boolean graph relating the derived input and output Boolean expressions using the Boolean operators: AND, OR, NAND, NOR, NOT. From this graph, or otherwise, a decision (binary truth) table representing the logical relationships between causes and effects is produced.

Unit Testing

ExamplePOS machine Condition 1 (C1) = deposit covers desired amountCondition 2 (C2) = overdraft covers desired amountCondition 3 (C3 = pin okAction1 (A1) = provide cashAction2 (A2) = suspend accountAction3 (A3) = issue information letter

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Cause-Effect Graphing(2)If the deposit or overdraft covers the desired amount, and the PIN is ok the POS provides cash. If the pin is not ok or the deposit is insufficient the account is suspended. An information letter is issued if overdraft does not cover desired amount, or pin is not ok.

Unit Testing

The code graph is then recast in terms of a decision table. Each column of the decision table is a rule.* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Cause-Effect Graphing(3)Following decision table corresponds to this example:

C1 deposit covers desired amountC2 overdraft covers desired amountC3 pin okA1 provide cashA2 suspend accountA3 issue letter1TTTTFF2TTFFTT3TFTTFT4TFFFTT5FTTTFT6FTFFTT7FFTFTT8FFFFTT

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Cause-Effect Graphing(4)MeasurementCoverage items are rules, where each rule represents a unique possible combination of inputs that have been expressed as Booleans.

These test cases provide 100% Cause-Effect coverage for our example, also from this table you can observe the major drawback of this technique, it doesnt say anything about the quality of the chosen inputs:

Test CaseInputExpected resultpin okdesired amountcurrent balanceoverdraft limit1T10010001000provide cash2F10010001000suspend, issue letter3T100100090provide cash, issue letter4F100100090suspend, issue letter5T10001000provide cash, issue letter6F10001000suspend, issue letter7T100090suspend, issue letter8F100090suspend, issue letter

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGFunctional - Syntax TestingDesignSyntax testing is a black boxtesting technique for protocol implementation. First step is to specify the syntax for the Protocol in a convenient notation (formal meta-language). Second step is to apply mutations to the syntactic elementsLast step is to modify the grammar in order to produce aberrant test vectors.

MeasurementNo coverage measure is defined for syntax testing.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Statement Testing(1)DesignStatement testing uses a model of the source code which identifies statements as either executable or non-executable. Test cases shall be designed to exercise executable statements.It answers to the question: Has each line of the source code been executed?

MeasurementCoverage items are executable statements in the source code.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Statement Testing(2)

If we consider following C code, any test case with b true will provide 100% statement coverage. Notice that it's not necessary to exercise b=FALSE for achieving full statement coverage.

a;if (b) { c;}d;

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch/Decision TestingDesignBranch testing requires a model of the source code which identifies decisions and decision outcomes. A decision is:a conditional transfer of control from any statement to any other statement in the unit under test;an unconditional transfer of control from any statement to any other statement in the unit except the next statement;when a unit has more than one entry point, a transfer of control to an entry point of the unit.Typical decisions are found in loops and selections. Each possible transfer of control is a decision outcome.

MeasurementCoverage item is branch / decision:

100%decision coverage 100% branch coverage (only for 100%)

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch/Decision Testing(2)

If we consider following C code and we mark with B the blocks and with D the decisions.Next step will be to create a control flow graph by making each basic block a node and drawing an arc for each possible transfer of control:

B1->B2; B2->B3; B2->B4; B4->B5; B4->B8; B5->B7; B5->B6; B3->B9; B8->B9; B7->B9; B6->B9

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch/Decision Testing(3)In this graph: Branches are the arrows, so we have 11 branches. Decisions are the blocks with more than one exit point, so we have 6 decisions

Example- Test input for: B1->B2->B3->B9 provides 3/11(27%) branch coverage and 1/6 (16,6%) decision coverage.- Test input forB1->B2->B4-> B5->B6->B9 provides 5/11 (45%) branch coverage and 3/6 (50%) decision coverage

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Data Flow Testing(1)DesignVariable occurrences in the unit under test, are split in two categories:Definitions are variable occurrences where a variable is given a new valueUses are variable occurrences where a variable is not given a new value:Predicate uses (P-uses) occur in the predicate portion of a decision statement such as while .. do, if .. then .. else, etc. Calculation uses (C-uses) are all others, including variable occurrences in the right hand side of an assignment statement, or an output statement.Test cases shall be designed to execute control flow paths between definitions and uses of variables.Measurement For the purposes of coverage calculations a definition-use pair is defined as a simple sub path between a definition of a variable and a use of that variable and coverage is calculated using:N = Number of exercised definition-use pairs from distinct variable definitionsT = Number of variable definitionsCoverage = (N/T) * 100%

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Data Flow Testing(2)For this technique we will use an equation solving function as example:

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Data Flow Testing(3)Next we create a table and assign a value for each variable use (definition, c-use, p-use)

linecategorydefinitionc-usep-use1a,b,c23da,b,c456d7complex89complex10complex11r1a,b,d12r2a,b,d13r1,r2

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Data Flow Testing(4)Next step is to identify definition-use pairs (du-pairs) for all variables. This is done by finding links from each entry in the definition column to each entry for that variable in the c-use or p-use column.

du-usecategoryc-usep-use1->3a,b,c1->11a,b1->12a,b3->6d3->11d3->12d4->13r1,r27->10complex9->10complex

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Data Flow Testing(5)From previous table can be created several sets of test cases to achieve definition uses, c-uses, p-uses and All-uses coverage.In order to achieve 100% All-uses coverage for all variables must be exercised at least one sub path from variable definition to every use of that definition. This test cases cover this scenario:

TCAll-uses / All du-pathsinputoutputvariable(s)d-u pair subpathabccomplexr1r21a,b,c1->31,3111Tnot definednot defined2a,b1->111,3,6,9,10,11121F-1-13a,b1->121,3,6,9,10,11,12121F-1-14d3->63,6111Tnot definednot defined5d3->113,6,9,10,11121F-1-16d3->123,6,9,10,11,12121F-1-17r1,r24->134,6,9,10,11,12,13121F-1-18complex7->107,10111Tnot definednot defined9complex9->109,10121F-1-1

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch Condition Testing (1)Branch Condition Testing requires a model of the source code which identifies decisions and the individual Boolean operands within the decision conditions. A decision is an executable statement which may transfer control to another statement depending upon the logic of the decision statement. A decision condition is a Boolean expression which is evaluated to determine the outcome of a decision. Typical decisions are found in loops and selections.Test cases shall be designed to exercise individual Boolean operand values within decision conditions.

Branch Condition Coverage is only defined for units with one entry point.

Measurement Coverage items are Boolean operand values within decision conditions.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch Condition Testing (2) ExampleOnly for simplicity a, b, c are Boolean operators, in real live most often a will be a expression like X>Y. Branch Condition Coverage would require all Boolean operands to be evaluated both TRUE and FALSE, this means that with only two test cases we can achieve 100% Branch Condition Coverage.Greatest disadvantage of Branch Condition Coverage is that it can often be achieved without testing both the TRUE and FALSE branches of the decision, for example above test cases test only TRUE decision branch.

TCabc1truefalsefalse2falsetruetrue

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch Condition Combination (1)Test cases shall be designed to exercise combinations of Boolean operand values within decision conditions.Measurement

Coverage items are unique combinations of the set of Boolean operand values within each decision condition.

Unit Testing

Example* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Branch Condition Combination (2) Branch Condition Combination Coverage would require all combinations of Boolean operands to be evaluated both TRUE and FALSE, this means that in order to achieve 100% Branch Condition Coverage we will need 23 = 8 test cases.Greatest disadvantage of Branch Condition Combination Coverage is that the test cases number needed for achieving full coverage increases exponentially. In real live this rapidly becomes unachievable.

TCabc1falsefalsefalse2falsefalsetrue3falsetruefalse4falsetruetrue5truefalsefalse6truefalsetrue7truetruefalse8truetruetrue

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Modified Condition Decision(1)Test cases shall be designed to demonstrate that Boolean operands within a decision condition can independently affect the outcome of the decision.

MeasurementCoverage items are Boolean operand values within decision conditions.Modified Condition Decision Coverage is only defined for components with one entry point.

Unit Testing

Example* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Modified Condition Decision(2) Modified Condition Decision requires test cases to show that each operand can independently affect the outcome of the decision. We will mark test cases where each operand can independently affect the outcome with:* for a, & for b and $ for c).

For achieving full Modified Condition Decision Coverage is required a minimum of n+1 test cases, and a maximum of 2n test cases, (where n is the number of Boolean operands within the decision condition).*&&$$*

TCabcOut1falsefalsefalsefalse2falsefalsetruefalse3falsetruefalsefalse4falsetruetruetrue5truefalsefalsetrue6truefalsetruetrue7truetruefalsetrue8truetruetruetrue

TCabcOut1falsefalsetruefalse3falsetruefalsefalse2falsetruetruetrue3truefalsetruetrue

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Linear Code Sequence and Jump [LCSAJ] (1)LCSAJ testing requires a model of the source code which identifies control flow jumps (where control flow does not pass to a sequential statement). LCSAJ is defined by a triple, conventionally identified by line numbers in a source codelisting: the start of the linear code sequencethe end of the linear code sequenceand the target line to which control flow is transferred.MeasurementCoverage items are LCSAJs

Unit Testing

Structural - Linear Code Sequence and Jump [LCSAJ] (2)Example for simplicity there will be no validation on input data. * / ALEXANDRU LUPU / 12.11.2010 Continental AG

Branches5->6not eof 5->16eof Jump7->8for loop execution factor13for loop to endJump8->9requires if condition to be true8->12requires if condition to be falseJump12->7further iteration of the for loopJump12->13requires for loop to be exhausted13->14requires prime to be true13->15requires prime to be falseJump16->5allways take placeJump

Unit Testing

From previous list we can find the LCSAJ start points. These are:the start of the component: line 3 lines to which control flow can jump from other than the preceding line: lines 5, 7, 8, 12 and 13.final of the component: line 17

This means that LCSAJs starting from line 3 are: 3->5->17; 3->7->13; 3->8->12; 3->12->7; 3->13->15; 3->16->5. Similar we can compute LCSAJs for the other lines

Notice that one LCSAJ is not one branch, or one jump, for example LCSAJ 2->16->5 requires 5 branches to execute in sequence: 3->5, 5->6, 7->8, 8->9, 12->13, 13->14 and a jump, 16->5

* / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Linear Code Sequence and Jump [LCSAJ] (3) in the end we will have something like this:

Branches5->6not eof 5->16eof Jump7->8for loop execution factor13for loop to endJump8->9requires if condition to be true8->12requires if condition to be falseJump12->7further iteration of the for loopJump12->13requires for loop to be exhausted13->14requires prime to be true13->15requires prime to be falseJump16->5allways take placeJump

LCSAJ#Start lineFinishLineJumpLine13516237133381243127531315631657551685713958121051271151315125165137713147812157127167131517716518881219812720813152181652212127231213152412165251313152613165281717exit

Unit Testing

We will consider a test scenario of inserting one prime number 3, one non-prime number 10, and a special number 1. This scenario is executed in a single test without stopping the application.For this test scenario we can determine the LCSAJ coverage by marking in the previous table executed LCSAJs and compute in the end the number of executed LCSAJs / total number of LCSAJs * 100%.

Execution result should look like: * / ALEXANDRU LUPU / 12.11.2010 Continental AGStructural - Linear Code Sequence and Jump [LCSAJ] (4)*********LCSAJ coverage is 9/28*100%= 32.14%

LCSAJ#Start lineFinish LineJump LineExecuted13516237133381243127531315631657551685713958121051271151315125165137713147812157127167131517716518881219812720813152181652212127231213152412165251313152613165281717exit

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGOther Techniques - Random TestingRandom Testing uses a model of the input domain of the unit that defines the set of all possible input values. The input distribution (normal, uniform, etc.) to be used in the generation of random input values shall be based on the expected operational distribution of inputs.Where no knowledge of this operational distribution is available then a uniform input distribution shall be used.Test cases shall be chosen randomly from the input domain of the unit according to the input distribution.MeasurementNo coverage metric is defined for this test.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGThis method has been created after the practical observation that the simplest bugs in a program are generally triggered by a single input parameter. The next simplest category of bugs consists of those dependent on interactions between pairs of parameters, which can be caught with all-pairs testing (~70% coverage). Bugs involving interactions between three or more parameters are progressively less common and more expensive.

Other Techniques All pairs Testing (1) DesignAll-pairs testing or pairwise testing is a combinatorial software testing method that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. The number of tests is typically O(nm), where n and m are the number of possibilities for each of the two parameters with the most choices. MeasurementNo coverage metric is defined for this test.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGExample: From a total number of 27(33) testcases, all-pairs method extracts 9(3x3) testcases. This reduce test count function from exponential to linear increase.Other Techniques All pairs Testing (2)

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGTechniques Observations (1)There is no established consensus on which techniques and criteria are the most effective. The only consensus is that the selection will vary as it should be dependent on a number of factors such as criticality, application area, and cost.

It is good practice to use functional test case design techniques and structural test measurement techniques.

Functional techniques are effective at detecting errors of omission, while structural techniques can only detect errors of commission. So a test plan could typically require boundary value analysis to be used to generate an initial set of test cases, while also requiring 100% branch coverage to be achieved.

With test completion criteria only 100% it is possible to relate some of techniques in an ordering, where criteria are shown to subsume, or include, other criteria.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGTechniques Observations (2)Partial Ordering of Structural Test Coverage Criteria

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGTechniques Observations (3)Know the limitations of each test design technique:Unit tests can never prove the correctness of code!!A failing test may indicate that the code contains errors, but a succeeding test doesn't prove anything at all.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGWhat have we seen?Why unit testing?Unit & ModuleTest Design Techniques- Functional- Structural- Other techniques- Observations

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGQUESTIONS?Structure and functionality must be tested together.

Unit Testing

* / ALEXANDRU LUPU / 12.11.2010 Continental AGReferenceshttp://www.testingstandards.co.ukhttp://www.ibm.com/developerworks/library/j-test.htmlhttp://developer.apple.com/mac/library/documentation/DeveloperTools/Conceptual/UnitTesting/Articles/UTGuidelines.html#//apple_ref/doc/uid/TP40002172-BBCBGHCJhttp://geosoft.no/development/unittesting.html