ORTHOGONAL ARRAY TESTING

Prince Bhanwra [801031024]Prince Bhanwra [801031024]

Ravinder Dahiya [801031025]Ravinder Dahiya [801031025]

PRESENTED BYPRESENTED BY : :

Contents Software Testing Conventional Test Cases

Conventional Testing Issues Software Faults Why OATS..?? Terminology of OATS Examples OATS Advantages OATS Limitations Challenges Mixed Mode OA Available Test Automation Tools References

Thapar University, Patiala2

Testing

SOFTWARE TESTINGSOFTWARE TESTING

Software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test.

CONVENTIONAL / EXHAUSTIVE TESTINGCONVENTIONAL / EXHAUSTIVE TESTING

A test approach in which the test suite comprises all combinations of input values and preconditions.

Thapar University, Patiala3

Conventional Test Cases

Thapar University, Patiala4

Example:If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue).The possible combinations in conventional test cases would be 27 i.e. 33

Conventional Testing Issues

Conventional Test Cases: Variables:3

Input: 4

Possible cases: 81=34

Variables: 3

Input: 5

Possible Cases: 243 = 35

… Variables: 5

Input: 5

Possible Cases: 3125 = 55

Thapar University, Patiala5

Orthogonal Test Cases Variables:3

Input: 4

Possible cases: 9 Variables: 3

Input: 5

Possible Cases: 11 … Variables: 5

Input: 5

Possible Cases: 21

Software Faults

Thapar University, Patiala6

1. Region Faults.2. Isolated Faults.

Orthogonal Array based testing is highly effective for the detection of region faults with a relatively small number of tests.

Why Orthogonal Array Testing (OATS)..?? Systematic, statistical way to test pair-wise interactions. Interactions and integration points are a major source of

defects. Most defects arise from simple pair-wise interactions.

“When the background is blue and the font is Arial and the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.”

Exhaustive testing is impossible. Execute a well-defined, concise set of tests that are likely

to uncover most (not all) bugs. Orthogonal approach guarantees the pair-wise coverage

of all variables.

Thapar University, Patiala7

Terminology for working with OA’sOA’s are commonly represented as :LRuns(LevelsFactors) or OA(Runs, Factors, Levels, Strength) or

OAλ(Runs(N); Factors(k), Levels(v), Strength(t))

is an N × k array on v symbols such that every N × t sub-array contains all tuples of size t from v symbols exactly λ times.

Runs (N) – Number of rows in the array, which translates into the number of Test Cases that will be generated.Factors (k) – Number of columns in the array, which translates into the maximum number of variables that can be handled by the array.Levels (v) – Maximum number of values that can be taken on by any single factor.Strength (t) – The number of columns it takes to see all the possibilities equal number of times.

� λ -1 for software testing and is often omitted

Thapar University, Patiala8

Example 1

Thapar University, Patiala9

A Web Page has three distinct sections (Top, Middle, Bottom) that can be individually shown or hidden from user

No.of Factors=3 (Top,middle,Bottom)

No.of Levels =2 (Hidden or shown)

Array Type =L4(2 3) or OA(4,3,2,2)

If we go for exhaustive testing we need :2 x 2 x 2 = 8 Test Cases

OA(Runs, Factors, Levels, Strength)

Example 1 (contd..)

Thapar University, Patiala10

Fixed Level Array: L4(2 3)

F1 F2 F3

Run1 0 0 0

Run 2 0 1 1

Run 3 1 0 1

Run 4 1 1 0

L423 – OA with 4 Runs3 factors with 2 levels

Top Middle Bottom

Test 1 Hidden Hidden Hidden

Test 2 Hidden Visible Visible

Test 3 Visible Hidden Visible

Test 4 Visible Visible Hidden

The Four Test Scenarios (4 Vs. 8)

1 - Display the home page and hide all sections. 2 - Display the home page and show all but the Top section.3 - Display the home page and show all but the Middle section.4 - Display the home page and show all but the Bottom section.

Example 2

Thapar University, Patiala11

Example:If we have three variables (A,B,C), each can have 3 values say (Red, Green, and Blue).The possible combinations in OATS test cases would be 9.

OATS advantage to select a test set:

Guarantees testing the pair-wise combinations of all the selected variables.

Creates an efficient and concise test set with many fewer test cases than testing all combinations of all variables.

Creates a test set that has an even distribution of all pair-wise combinations.

Exercises some of the complex combinations of all the variables.

Is simpler to generate and less error prone than test sets created by hand.

Thapar University, Patiala12

OATS basic fault model: Interactions and integrations are a major source of defects. Most of these defects are not a result of complex

interactions such as “When the background is blue and the font is Arial and

the layout has menus on the right and the images are large and it’s a Thursday then the tables don’t line up properly.”

Most of these defects arise from simple pair-wise interactions such as “When the font is Arial and the menus are on the right

the tables don’t line up properly.”

Thapar University, Patiala13

OATS basic fault model (contd..): With so many possible combinations of

components or settings, it is easy to miss one. Randomly selecting values to create all of the pair-

wise combinations is bound to create inefficient test sets and test sets with random, senseless distribution of values.

It covers 100% (9 of 9) of the pair-wise combinations, 33% (9 of 27) of the three-way combinations and 11% (9 of 81) of the four-way combinations.

Thapar University, Patiala14

The Challenges The number of tests may increase rapidly as the degree of

interactions increases ‰ Test generation algorithms must be more sensitive in

terms of both time and space requirements ‰ The need for test automation becomes even more

serious. Impractical to manually execute and inspect the results

of a large number of test runs However, on the Web there are pre-calculated

orthogonal array tables for certain numbers of variables and values.

Thapar University, Patiala15

Mixed Mode OA

Thapar University, Patiala16

Available Test Automation Tools

Thapar University, Patiala17

1.CATS (Constrained Array Test System) *)

[Sherwood] Bell Labs.

2.OATS (Orthogonal Array Test System) *) [Phadke] AT&T

3.AETG Telecordia

Web-based, commercial

4. IPO (PairTest) *) [Tai/Lei]

5. TConfig [Williams] Java-applet

6.TCG (Test Case Generator) *) NASA

7.AllPairs Satisfice

Perl script, free, GPL

8.Pro-Test SigmaZone

GUI, commercial

9.CTS (Combinatorial Test Services) IBM

Free for non-commercial use

References http://csrc.nist.gov/groups/SNS/acts/documents/ipo-nist.pdf http://csrc.nist.gov/groups/SNS/acts/documents/aberdeen.ppt http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-system

-design-summer-1998/lecture-notes/l3_matrix_experiments4.pdf http://ocw.mit.edu/courses/aeronautics-and-astronautics/16-881-robust-

system-design-summer-1998/lecture-notes/l8_orth_arrays.pdf http://delivery.acm.org/10.1145/1290000/1283384/p1-abam.pdf?

key1=1283384&key2=5753427921&coll=DL&dl=ACM&CFID=9321178&CFTOKEN=33991028

http://www.developsense.com/pairwiseTesting.html http://learnsoftwareprocesses.com/2007/08/21/orthogonal-array-testing-

strategy-oats/comment-page-1/#comment-7394 http://www.vietnamesetestingboard.org/zbxe/?document_srl=219676 http://www.pairwise.org/tools.asp

Thapar University, Patiala18

Queries..?Queries..?

Thapar University, Patiala19

Thank you !!Thank you !!

Thapar University, Patiala20