pairwise testing comes of age - testcover.comtestcover.com/pub/background/stareast2008s.pdfthis talk...

Testcover.com

Copyright © 2008 Testcover.com, LLC. All rights reserved.

Pairwise Testing Comes of Age

George B. Sherwood, Ph.D.

Thursday May 8, 1:30 p.m.T14 Pairwise Testing

Testcover.com


Pairwise Testing Comes of Age1920

Design of Experiments1940 Orthogonal Arrays

OAs for Manufacturing1960

Covering Arrays1980 OAs for Software

OATS, CATS, AETG Pairwise testing2000 … …

Many more tools Much more testing2020

PresenterPresentation NotesThis talk is about the development of pairwise testing. I plan to tell you about my personal experience with a couple of the early tools – why I got involved, what worked, and what didn’t.There are two test design examples we’ll be going through. They illustrate topics that in my view are not discussed and understood well enough. We’ll look at test factor constraints in both examples, but for two different reasons. In the second example we’ll also see how to use expected results to insure that all the pairs are tested as intended.Over the years different techniques have been used to generate pairwise tests. There are various algorithms for searches and for constructing arrays, and they involve tradeoffs. I’ll go through some examples of these, including a few math tricks. My purpose here is to illustrate the tradeoffs for testers. Just as it helps to understand the differences in how cars work when you’re choosing one to drive, it’s useful to think about the different ways to generate pairwise test cases.

Sheet1

1920

Design of Experiments

1940Orthogonal Arrays

OAs for Manufacturing

1960

Covering Arrays

1980OAs for Software

OATS, CATS, AETGPairwise testing

2000……

Many more toolsMuch more testing

2020

Sheet2

Sheet3

Testcover.com


Évariste GaloisFrench mathematician, studied abstract •algebra & group theoryRadical Republican during the reign of •Louis PhilippeShort life with tragic ending:•

• 1829• Father committed suicide • Failed entry to École

Polytechnique• 1830

• Expelled from École Normale for criticizing director

• 1831• Imprisoned for wearing

illegal uniform • 1832

• Rejected by Stéphanie-Felice du Motel

• Shot in duel at age 20

PresenterPresentation NotesHas anyone here heard of Évariste Galois?He was a French mathematician and radical Republican. He was involved in the political turbulence after Napoleon’s defeat, during the restoration of the monarchy. Galois made an important contribution to the math behind pairwise testing. But his story is remarkable for the amount of tragedy during the last three years of his life.His father, the mayor of Bourg-la-Reine, hanged himself as a result of a scandal caused by the local priest who forged the mayor’s name. Then during the July Revolution in 1830 there was rioting in Paris, and Galois’ school director locked up the students to prevent their joining in. Galois’ criticism of the director got him expelled from school.Galois joined a Republican branch of the militia, the Artillery of the National Guard. But King Louis-Phillipe felt it was a threat to the throne, and he abolished the group. Then on Bastille Day in 1831, Galois headed a protest wearing his artillery uniform, armed with a rifle, several pistols, and a dagger. He was arrested and sentenced to six months in prison. He fell in love and was rejected by Mlle. Stéphanie-Felice du Motel, the daughter of a prison physician. Then a month after his release from prison, for reasons that are still unclear, he was shot in a duel, and died the next day.Galois’ mathematics was not understood by his contemporaries, and he had trouble publishing his papers. So what does Galois have to do with pairwise testing?We’ll see that in a minute.____________http://en.wikipedia.org/wiki/Galois

Testcover.com


21st Century Testing• Increasing dependency on software,

networks and distributed applications• Increasing expectation and need for

quality• Facts of life:

• More features and complexity• Shorter development cycles• More reuse of components

PresenterPresentation NotesWe live in the 21st century. Every day we become more dependent on software, networks and distributed applications. Our shopping, banking, appliances and health care are increasingly automated by systems that must work correctly. But along with this progress there are software problems causing lost productivity, power outages, recalled cars and even personal injury, because systems do not always work correctly.Our test organizations, and our tools and processes, have not kept up with the rapid proliferation of new technologies and applications. Our expectations for more features in shorter development cycles have challenged our ability to manage the complexity of the work. Testers need effective ways to deal with the complexity and achieve quality.____________http://catless.ncl.ac.uk/risks

Testcover.com


Why Pairwise Testing?Statistical coverage to find faults•

̶ Improved quality

Small number of test cases•̶ Reduced cost

PresenterPresentation NotesToday I’ll be talking about pairwise testing. This is a way to generate test cases that can be more effective than random selection or guessing. There are two reasons for this improvement. First, pairwise test cases have statistical properties to make them better able to find faults. Second, pairwise testing can result in fewer test cases to achieve this statistical coverage. Better ability to find faults can lead to improved quality; fewer test cases mean lower costs.I want to be clear that we’re not talking about code coverage here. Code coverage can be a useful part of unit testing, but pairwise testing usually comes later. Typically pairwise testing is used two ways: to select test configurations, or to select input data values to verify functionality.

Testcover.com


Pairwise Testing Example• Test 3 new applications on a networked

computer • 4 test configuration factors with 3 values

• 34=81 configurations in total – too many• 3 configurations cover all values individually

– but only a few interactions

Op System Connection Browser ApplicationXP Dial-up Int Explorer App1OS X DSL Netscape App2Linux LAN Firefox App3

PresenterPresentation NotesLet’s look at a test configuration example.Here we have three new applications to be tested on a networked computer. We need to specify which test configurations will be used to run a suite of system tests. We have determined that there are four test factors to be varied during the tests – the operating system, the network connection type, the browser, and the application. The table shows the required values.Because we have 4 factors with 3 values, there are 3 x 3 x 3 x 3 = 81 possible configurations. We don’t have the time or the resources for 81 configurations; it’s not practical to test all of them. We can pick out 3 configurations to test all the values individually. For example, we can use the 3 rows of this table. But what if there are software faults which happen only when we use two values of different factors together, like Dial-up with Application 2? Our 3 configurations will test only 33% of the pairwise interactions. Is it feasible to choose a few configurations that cover all the pairwise interactions? That is exactly what pairwise testing aims to do.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)

+012Op SystemConnectionBrowserApplication

0012XPDial-upInt ExplorerApp1

1120OS XDSLNetscapeApp2

2201LinuxLANFirefoxApp3

test def


00120 XP0 Dial-up0 Int Explorer0 App1

11201 OS X1 DSL1 Netscape1 App2

22012 Linux2 LAN2 Firefox2 App3

test conf


0012XPDial-upInt ExplorerApp 1

1120XPDSLNetscapeApp 2

2201XPLANFirefoxApp 3

OS XDial-upNetscapeApp 3

OS XDSLFirefoxApp 1

OS XLANInt ExplorerApp 2

LinuxDial-upFirefoxApp 2

LinuxDSLInt ExplorerApp 3

LinuxLANNetscapeApp 1

test conf (2)






OS XDSLFirefoxApp 1





test conf (3)


0012XPDial-upNetscapeApp 1

1120XPDSLFirefoxApp 2

2201OS XDSLNetscapeApp 3

OS XLANFirefoxApp 1



XPLANInt ExplorerApp 3

OS XDial-upFirefoxApp 2

LinuxDSLNetscapeApp 1

XPDial-upInt ExplorerApp 2

XPDSLInt ExplorerApp 1

4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Pairwise TestingIncludes all pairs of interactions •between test factorsProvides a small set of test cases • – a practical alternative to testing all combinationsStarted almost • 20 years agoIncreasingly popular way to improve •quality and reduce cost

PresenterPresentation NotesThere have been studies that indicate most software faults are caused by individual test values or by pairs of them. For example, in 2001 Wallace and Kuhn studied 15 years of medical device recall data and found that almost all of the failures they could analyze were caused by two or fewer conditions.Pairwise testing is a way to design tests that include all pairs of factor values. The technique can be used to select physical test configurations, data input, environmental factors, and more. The ideas behind pairwise testing came out of the study of experimental design. About 20 years ago the first test case generators emerged. Since then pairwise testing has become an increasingly popular way to improve quality and reduce cost.What does Galois have to do with this?As engineers we learn about fields – electric fields, gravitational fields. The real numbers are a field. That means they follow simple rules of algebra we learned in high school. These examples of fields are infinite, but it is possible to have finite fields also. The math of finite fields is important for generating pairwise test cases. And it turns out that finite fields are named after Évariste Galois.____________http://www.stickyminds.com/getfile.asp?ot=XML&id=6449&fn=XDD6449filelistfilename1.pdfhttp://en.wikipedia.org/wiki/Experimental_design

Testcover.com


Galois Field Example• Modulo 3 arithmetic – GF(3)• Usual + or x, divide by 3 & take remainder

• Use GF(3) tables to make orthogonal array

+ 0 1 2 x 0 1 20 0 1 2 0 0 0 01 1 2 0 1 0 1 22 2 0 1 2 0 2 1

PresenterPresentation NotesHere’s an example of a Galois field with 3 elements, which is what is meant by GF(3). The tables are for addition and multiplication in the finite field. In this case it’s just modulo 3 arithmetic. The way that works is you do the usual addition or multiplication. Then you divide by 3 and take the remainder. For example 2 + 1 = 3; divide by 3 and the remainder is 0. 2 x 2 = 4; divide by 3 and the remainder is 1.We can make an orthogonal array for our networked computer example using this Galois field. What we’re going to do is take the 0 column from the addition table and replace all the 0s in the multiplication table with it. Then we’ll do the same replacement using the 1 and 2 columns. So the 3-by-3 multiplication table will become an array with 9 rows and 3 columns.____________http://www.research.att.com/~njas/doc/OA.htmlhttp://projecteuclid.org/DPubS/Repository/1.0/Disseminate?view=body&id=pdf_1&handle=euclid.aoms/1177729387

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Orthogonal Array OA(9;33)Replace each element in the x table with •the corresponding column from the + table

+ 0 1 2 x 0 1 20 0 1 2 0 0 0 01 1 2 0 1 1 12 2 0 1 2 2 2

1 0 1 21 2 02 0 1

2 0 2 11 0 22 1 0

PresenterPresentation NotesNow what used to be the multiplication table is an orthogonal array. You can see where the columns from the addition table have been placed. The notation OA(9;33) indicates the array has 9 rows or test cases. The base 3 says there are factors with 3 values, and the exponent says there are 3 of these factors. The 33 indicates there are 27 possible test cases here, if we consider all possible combinations.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Orthogonal Array OA(9;34)• Add one column which pairs a different

value with every 3 rows0 0 0 00 1 1 10 2 2 21 0 1 21 1 2 01 2 0 12 0 2 12 1 0 22 2 1 0

PresenterPresentation NotesWe actually can get one additional factor for the array by placing one of the 3 values in each subarray of 3 rows. Now we have an orthogonal array which has 4 factors with 3 values each. This means if you pick any two columns of the array, you’ll find all nine pairs of values. My point in going through this construction is that we don’t need to do much processing to generate an orthogonal array. We didn’t have to do any searching at all. So if an orthogonal array works as a test template, it should be easy for a tool to generate it.That’s enough math for now. Let’s see how this array relates to the test example.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def


00120=>XP0=>Dial-up0=>Int Explorer0=>App 1

11201=>OS X1=>DSL1=>Netscape1=>App 2

22012=>Linux2=>LAN2=>Firefox2=>App 3

test conf






OS XDSLFirefoxApp 1





Sheet2

Sheet3

Testcover.com


Networked Computer Example4 • test configuration factors with 3 valuesReplace numbers in OA(• 9;34) template with test factor values

Op System Connection Browser Application0 XP 0 Dial-up 0 Int Explorer 0 App11 OS X 1 DSL 1 Netscape 1 App22 Linux 2 LAN 2 Firefox 2 App3

PresenterPresentation NotesIn our networked computer example we were looking for a few test cases which would cover all pairs of values. Coincidentally the numbers of test factors and their values match our orthogonal array perfectly.So how can we use these values to select the test configurations? Simply replace the number values in the orthogonal array template with the corresponding test factor values. For the operating system, 0 is replaced with XP; for the connection type 1 is replaced with DSL…

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Networked Computer Example• Only 9 configurations are needed to cover

all 54 pairs of values using OA(9;34)Op System Connection Browser Application

XP Dial-up Int Explorer App 1XP DSL Netscape App 2XP LAN Firefox App 3

OS X Dial-up Netscape App 3OS X DSL Firefox App 1OS X LAN Int Explorer App 2Linux Dial-up Firefox App 2Linux DSL Int Explorer App 3Linux LAN Netscape App 1

PresenterPresentation NotesWhen the replacement is done, we have 9 test configurations. It’s easy to see that XP is paired with all values of the other factors in the first 3 configurations. We also see that Dial-up is paired with all 3 browsers and all 3 applications.In total, there are 54 pairs covered in the 9 configurations. Each configuration includes 6 pairs, and none of them is repeated.We’ll come back to this example.You might be wondering where orthogonal arrays came from, and how software testers started using them.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Design of ExperimentsRonald A. Fisher analyzed data •from crop yield experimentsDeveloped new statistical methods to •study experimental data Pioneered the mathematical principles of •experimental design

PresenterPresentation NotesRonald Fisher was an English statistician and biologist. In the 1920s he worked at the Rothamsted Experimental Station analyzing data from crop yield experiments. For example, experimenters would test different kinds of fertilizer to see what worked best. While there Fisher created the theory and methods to do the analysis. He invented statistical techniques for the analysis of variance, maximum likelihood and much more. He literally wrote the book on The Design of Experiments.____________http://en.wikipedia.org/wiki/Ronald_Fisher

Testcover.com


Orthogonal Arrays

• Orthogonal Array designs are used in agricultural & manufacturing experiments, clinical trials, market research…

• Results can show effects of factors and how they interact

• Calyampudi Radhakrishna Rao introduced Orthogonal Arrays

PresenterPresentation NotesCalyampudi Radhakrishna Rao was Fisher’s student at Cambridge. Rao studied and introduced orthogonal arrays in the 1940s. Currently he is Professor Emeritus of Statistics at Penn State.Over the past 60 years orthogonal arrays have been used more and more, to design experiments for agriculture, manufacturing, clinical trials for new drugs, and market research. Orthogonal arrays are powerful tools for discovering the effects of test factors and how they interact.____________http://en.wikipedia.org/wiki/C._R._Raohttp://www.research.att.com/~njas/doc/OA.html

Testcover.com


OAs for ManufacturingGenichi Taguchi adapted Fisher• ’s experimental design to improve quality of manufactured goods

• Used OAs to select test parameters in creative ways

• Worldwide influence in improving manufactured quality

PresenterPresentation NotesIn the 1950s Genichi Taguchi began to work on improvements to quality and reliability in manufacturing. He collaborated and consulted widely, working with Fisher for a time at the Indian Statistical Institute.Taguchi’s use of experimental design was innovative and sometimes controversial. Overall, his ideas have had an extensive influence in improving manufactured quality.____________http://en.wikipedia.org/wiki/Genichi_Taguchi

Testcover.com


OAs for Software Test• Keizo Tatsumi, Robert Mandl & others

applied these ideas to software testing• Taguchi visited Bell Labs and

collaborated with Madhav Phadke• Orthogonal Array Test System (OATS)

offered statistical coverage where exhaustive testing was not possible

PresenterPresentation NotesBy the 1980s people began to work out the experimental design ideas for testing software. Keizo Tatsumi at Fujitsu described the use of orthogonal arrays and covering arrays to generate test cases. Robert Mandl, at Softech, used orthogonal Latin squares to design compiler tests.Taguchi visited Bell Labs and began a collaboration with Madhav Phadke. Taguchi’s methods gained acceptance at AT&T and inspired Bob Browlie, Jim Prowse and Madhav Phadke to come up with the Orthogonal Array Test System (OATS).____________http://www.pairwise.org/docs/icqc87.pdfhttp://portal.acm.org/citation.cfm?id=4375&jmp=review&dl=GUIDE&dl=ACMRobert Brownlie, et al. “Robust Testing of AT&T PMX/StarMAIL Using OATS,” AT&T Technical Journal, Vol. 71, No. 3, May/June 1992, pp. 41-47.

Testcover.com


StarLAN and StarGROUPClient installation test•

101 • combinations of PCs and OS versions15 • network cards

Tried to use OATS to reduce number of •test configurationsOATS could not handle our complex •configuration constraints correctlyGenerated impossible tests, as in •previous example

PresenterPresentation NotesI was at Bell Labs then and had no idea any of this was going on. I was supervising a system test team responsible for a family of new local area network products. StarLAN hardware would be the first products to use Ethernet CSMA/CD protocols on telephone cables, at the speedy rate of 1 Mb/s. StarGROUP software would offer file sharing and printer sharing over the network, to improve office productivity and reduce costs.We had what we called a client installation test. It included installation and some operational tests, and it was here that our extensive configuration requirements hit us. The job was large and labor intensive. We had to support 101 combinations of PCs and operating system versions with 15 different network cards. It was then that I heard about OATS, as a possible way to reduce the number of test configurations.Unfortunately OATS didn’t work for us. We had complex configuration constraints which disallowed some of the test factor combinations. And some of the configurations that OATS generated were these impossible cases.Instead of trying to describe 1990 technology here, I’d like to illustrate the problem with the previous example. ____________http://testcover.com/pub/background/cats1.htm

Testcover.com


Networked Computer Example• Two of the 9 configurations from

OA(9;34) are invalid

Op System Connection Browser ApplicationXP Dial-up Int Explorer App 1XP DSL Netscape App 2XP LAN Firefox App 3

OS X Dial-up Netscape App 3OS X DSL Firefox App 1OS X LAN Int Explorer App 2Linux Dial-up Firefox App 2Linux DSL Int Explorer App 3Linux LAN Netscape App 1

PresenterPresentation NotesThe orthogonal array we used in the networked computer example paired the Internet Explorer browser with all three operating systems. But the only pair that is supported and represents a valid configuration is Windows XP with Internet Explorer. Our requirements don’t allow for old versions of Internet Explorer on OS X or third party implementations of Windows API. So the two configurations pairing Mac OS X with Internet Explorer and Linux with Internet Explorer cannot be included in the test plan.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


What to do?Skip the bad cases? Then for each case there •is no coverage for 5 good pairs, along with the invalid oneReplace values • ‘Linux’, ‘OS X’ or ‘Int Explorer’ to avoid bad pairs? Still miss good pairsSolution: Computer search for small number •of cases to cover all allowed pairs.

PresenterPresentation NotesWhat can we do to fix this?One idea is just to skip the bad configurations. The problem here is that each configuration from the orthogonal array covers six pairs. Each skipped configuration will miss 5 good pairs along with the one bad pair.We can try to replace one or more values in the bad configurations to make them valid, but generally this won’t work either. We still miss some of the valid pairs we want to include.The solution I used in 1990 was to build a new test case generator, one that would search for a small number of cases to cover only the allowed combinations.

Testcover.com


Networked Computer Example• 11 configurations cover 52 allowed pairs

Op System Connection Browser ApplicationXP Dial-up Netscape App 1XP DSL Firefox App 2

OS X DSL Netscape App 3OS X LAN Firefox App 1Linux Dial-up Firefox App 3Linux LAN Netscape App 2XP LAN Int Explorer App 3

OS X Dial-up Firefox App 2Linux DSL Netscape App 1XP Dial-up Int Explorer App 2XP DSL Int Explorer App 1

PresenterPresentation NotesIn the networked computer example, a computer search can find 11 configurations to cover the 52 allowed pairs. The two invalid pairs are not included.In 1990 we had to constrain the scope of the problem to avoid configurations we could not test. So the new tool then was named CATS – the Constrained Array Test System.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def (0)





test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


Constrained Array Test SystemSearch among allowed test cases to •handle constraintsFind next • “best” test case to minimize uncovered pairsSecondary result: fewer test cases than •OATS

PresenterPresentation NotesWhat CATS did was search among the allowed test cases to handle constraints of the system under test. It looked for a sequence of tests to minimize the number of uncovered pairs. And it did this simply by choosing the next “best” test case on the way to the minimum. This type of algorithm is sometimes called a greedy search. It takes advantage of the computer’s speed to select a small number of test cases.The motivation for CATS was to find test cases that were practical to use. But we also found that typically CATS generated fewer test cases than OATS. The reason for this was that CATS was looking for covering arrays, not orthogonal arrays.____________http://testcover.com/pub/background/cats2.htm

Testcover.com


Types of ArraysOrthogonal Arrays Covering Arrays

All pairs in equal numbers at least onceTest cases a few fewer typically

Fixed values OA(18;37) CA(14;37)Mixed values OA(12;3124) CA(6;3124)

PresenterPresentation NotesWhat’s the difference?An orthogonal array contains all pairs an equal number of times. This “balanced” property is useful for measuring the effects of factors and how they interact. However, for pass/fail results of software testing, we don’t need the arrays to be balanced. Instead, covering arrays contain all pairs at least once. They are a generalization of orthogonal arrays. So orthogonal arrays are covering arrays, but the reverse is not always true.Covering arrays can be smaller than orthogonal arrays because they don’t have to include pairs equally. For example, an orthogonal array with 7 factors of 3 values requires 18 test cases. A corresponding covering array has 14 test cases.For testing systems in the real world we need arrays that have factors with different numbers of values. These are called mixed arrays. We can see the size advantage for covering arrays here also. The mixed covering array has half the test cases that the mixed orthogonal array has.

Sheet1

Orthogonal ArraysCovering Arrays

All pairsin equal numbersat least once

Test casesa fewfewer typically

Fixed valuesOA(18;37)CA(14;37)

Mixed valuesOA(12;3124)CA(6;3124)

Sheet2

Sheet3

Testcover.com


Early Tools

Orthogonal ArrayTest System (OATS)

Constrained ArrayTest System (CATS)

Automatic EfficientTest Generator (AETG)

Bell Labs Bell Labs Bellcore (Telcordia)Internal tool Internal tool Commercial service

OA templateswith dependencies

Greedy search forCAs with constraints

Greedy search forCAs with constraints

PresenterPresentation NotesI’d like to wrap up this part of the discussion with a comparison of three of the earliest pairwise test tools. All of them came from organizations which had once been part of Bell Labs.OATS used orthogonal arrays to select test cases. It was used for several projects within AT&T.CATS used a greedy search algorithm for mixed covering arrays with constraints. It was used within AT&T, and after some study was dubbed a best practice.The Automatic Efficient Test Generator (AETG) was developed in Bellcore shortly after CATS. As far as I know, it was the first commercial tool. AETG also uses a greedy search for mixed covering arrays with constraints.____________http://aetgweb.argreenhouse.com

Sheet1

Orthogonal ArrayTest System (OATS)Constrained ArrayTest System (CATS)Automatic EfficientTest Generator (AETG)

Bell LabsBell LabsBellcore (Telcordia)

Internal toolInternal toolCommercial service

OA templateswith dependenciesGreedy search forCAs with constraintsGreedy search forCAs with constraints

Sheet2

Sheet3

Testcover.com


Pairwise Selection of Input DataIdentify equivalence classes•

Analyze functional requirements, use cases•Partition factor value combinations for equivalent •processing and expected results

Generate test cases for different classes separately•All pairs from each class go to same resulting state•Example: Don• ’t mix error combinations with normal combinations

PresenterPresentation NotesIn the networked computer example, we selected test configurations. We also can use pairwise test generation to select input data. When we do so, it is important to consider the states the system goes through as it executes a test scenario or use case. We need to group input values into equivalence classes, so they all lead to the same expected result. If we don’t do this, some of the pairs may lead to one result, while other pairs lead to a different result. If this happens, we may not be testing everything we intend to test.When we select input data, we want to see the system provide a normal result for a valid entry. We may also want to verify that it processes invalid input correctly, by going to a different, error processing state. If we simply mix all the input data together for our pairwise generator, we won’t know which pairs tested the normal result or which pairs tested the error handling.Our second example will illustrate this idea. We’re going to generate test cases for the interface to a web-based service. We’ll generate valid input for a normal result, as well as invalid data to look at error handling.

Testcover.com


Uncle Katzendogg’s Guaranteed Long-term Weather Forecasts

PresenterPresentation NotesThe service is Uncle Katzendogg’s Guaranteed Long-Term Weather Forecasts. We’re looking at the HTML form that provides the interface to the service. The user selects a month from the menu, enters the day of the month, the year and the zip code. When the forecast button is clicked, the data from the form are processed to return a weather forecast for the given date and zip code.

Testcover.com


Weather Forecast Example4 • test factors: Month, Day, Year, Zip code3 • sets of test cases leading to different states:

Valid dates with valid zip codes1.̶ Forecast returnedInvalid dates with valid zip codes2.̶ Date error returnedValid dates with invalid zip codes3.̶ Zip code error returned

PresenterPresentation NotesWe have 4 test factors:Month – We’ll use all 12 monthsDay of the month – We’ll use the first day, the 10th day and the last day for valid dates. For invalid dates we’ll look at one-off errors like May 32nd.Year – We’ll use 3 different years. One of them will be a leap year.Zip code – We’ll test with 2 zip codes that correspond to real places and 2 that are invalid.We need to generate input data for 3 different equivalence classes:Valid dates with valid zip codes, from which the forecast is returnedInvalid dates with valid zip codes, from which a date error is returnedValid dates with invalid zip codes, from which a zip code error is returned

Testcover.com


Weather Forecast Valid Input Factor Values

+ all good dates + feb last dayMonth: jan feb mar apr may jun

jul aug sep oct nov decMonth: feb

Day: 1 10 Day: 28Year: 2008 2009 2010 Year: 2009 2010

Zip code: 07722 32819 Zip code: 07722 32819+ long month last day + leap day

Month: jan mar may jul aug oct dec

Month: feb

Day: 31 Day: 29Year: 2008 2009 2010 Year: 2008

Zip code: 07722 32819 Zip code: 07722 32819+ short month last day

Month: apr jun sep novDay: 30

Year: 2008 2009 2010Zip code: 07722 32819

PresenterPresentation NotesThis slide shows the factor values we will give to the test case generator for valid input data – with good dates and good zip codes – for the first equivalence class. We’re going to constrain the factor values so the input data always will be valid. Each box in the slide lists factor values which can be used in a test case; any combination of values in a box is valid. Taken together, the 5 boxes specify all the valid factor values we want to use.In the first box we have all 12 months, days 1 and 10, the 3 years we’ll use, and 2 valid zip codes. Any combination of these values yields a good date and a good zip code. In the second box we have the long months paired with day 31, and the third box has short months with day 30. The last two boxes are for February with day 28 in a regular year and with day 29 in a leap year.

Weather forecast 1

Testcover.com

Results for George Sherwood, Testcover.com, LLC

Wed Jan 23 21:03:09 EST 2008

Weather Forecast Example

Request data:


Month

Day

Year

Zip code

#ok All good dates

jan feb mar apr may jun jul aug sep oct nov dec

1 10

2008 2009 2010

07722 32819

+ long month last day

jan mar may jul aug oct dec

31

2008 2009 2010

07722 32819

+ short month last day

apr jun sep nov

30

2008 2009 2010

07722 32819

+ feb last day

feb

28

2009 2010

07722 32819

+ leap day

feb

29

2008

07722 32819

#1. All good dates

TestMonthDayYearZip codeCombo

Case IDCountdown

12 Values6 Values3 Values2 Values168

ok1jan1200807722162

ok2jan10200932819156

ok3feb1201032819150

ok4mar10201007722144

ok5aug31200907722138

ok6oct31200832819132

ok7nov30200932819127

ok8jun30200807722122

ok9feb28200907722117

ok10mar1200932819113

ok11may10200832819109

ok12may31201007722105

ok13apr30201032819101

ok14feb2920080772297

ok15apr120080772294

ok16jun120093281991

ok17jul120080772288

ok18jul1020093281985

ok19aug1020083281982

ok20sep1020083281979

ok21dec1020083281976

ok22dec3120100772273

ok23sep3020090772270

ok24apr1020093281968

ok25jun1020100772266

ok26aug120103281964

ok27oct120080772262

ok28oct1020093281960

ok29nov1020083281958

ok30nov120103281956

ok31dec120093281954

ok32jul3120103281952

ok33jan3120103281950

ok34mar3120080772248

ok35feb2820103281946

ok36feb1020083281945

ok37may1020090772244

ok38may120103281943

ok39sep120093281942

ok40sep1020100772241

ok41oct1020103281940

ok42nov1020090772239

ok43feb2920083281938

Copyright © 2003-2007 Testcover.com, LLC. All rights reserved.

Weather forecast 1 input

+ all good dates+ feb last day

Month:jan feb mar apr may jun jul aug sep oct nov decMonth:feb

Day:1 10Day:28

Year:2008 2009 2010Year:2009 2010

Zip code:07722 32819Zip code:07722 32819

+ long month last day+ leap day

Month:jan mar may jul aug oct decMonth:feb

Day:31Day:29

Year:2008 2009 2010Year:2008

Zip code:07722 32819Zip code:07722 32819


Month:apr jun sep nov

Day:30

Year:2008 2009 2010

Zip code:07722 32819

Weather forecast 1 output

MonthDayYearZip codeMonthDayYearZip codeMonthDayYearZip code

jan1200807722jun1200932819nov1201032819

jan10200932819jul1200807722dec1200932819

feb1201032819jul10200932819jul31201032819

mar10201007722aug10200832819jan31201032819

aug31200907722sep10200832819mar31200807722

oct31200832819dec10200832819feb28201032819

nov30200932819dec31201007722feb10200832819

jun30200807722sep30200907722may10200907722

feb28200907722apr10200932819may1201032819

mar1200932819jun10201007722sep1200932819

may10200832819aug1201032819sep10201007722

may31201007722oct1200807722oct10201032819

apr30201032819oct10200932819nov10200907722

feb29200807722nov10200832819feb29200832819

apr1200807722

Testcover.com


Weather Forecast Valid Input Test Cases

Month Day Year Zip code Month Day Year Zip code Month Day Year Zip codejan 1 2008 07722 jun 1 2009 32819 nov 1 2010 32819jan 10 2009 32819 jul 1 2008 07722 dec 1 2009 32819feb 1 2010 32819 jul 10 2009 32819 jul 31 2010 32819mar 10 2010 07722 aug 10 2008 32819 jan 31 2010 32819aug 31 2009 07722 sep 10 2008 32819 mar 31 2008 07722oct 31 2008 32819 dec 10 2008 32819 feb 28 2010 32819nov 30 2009 32819 dec 31 2010 07722 feb 10 2008 32819jun 30 2008 07722 sep 30 2009 07722 may 10 2009 07722feb 28 2009 07722 apr 10 2009 32819 may 1 2010 32819mar 1 2009 32819 jun 10 2010 07722 sep 1 2009 32819may 10 2008 32819 aug 1 2010 32819 sep 10 2010 07722may 31 2010 07722 oct 1 2008 07722 oct 10 2010 32819apr 30 2010 32819 oct 10 2009 32819 nov 10 2009 07722feb 29 2008 07722 nov 10 2008 32819 feb 29 2008 32819apr 1 2008 07722

PresenterPresentation NotesHere are the test cases for valid input data. You can see that January is paired with day 1, 10 and 31, with year 2008, 2009, 2010, with both zip codes, etc. I’d like to point out February 29, 2008, a leap day, is included. All the dates and zip codes are valid, so our expected result is a weather forecast.

Weather forecast 1

Testcover.com


Wed Jan 23 21:03:09 EST 2008


Request data:


Month

Day

Year

Zip code

#ok All good dates

jan feb mar apr may jun jul aug sep oct nov dec

1 10

2008 2009 2010

07722 32819

+ long month last day


31

2008 2009 2010

07722 32819


apr jun sep nov

30

2008 2009 2010

07722 32819

+ feb last day

feb

28

2009 2010

07722 32819

+ leap day

feb

29

2008

07722 32819

#1. All good dates

TestMonthDayYearZip codeCombo

Case IDCountdown


ok1jan1200807722162

ok2jan10200932819156

ok3feb1201032819150

ok4mar10201007722144

ok5aug31200907722138

ok6oct31200832819132

ok7nov30200932819127

ok8jun30200807722122

ok9feb28200907722117

ok10mar1200932819113

ok11may10200832819109

ok12may31201007722105

ok13apr30201032819101

ok14feb2920080772297

ok15apr120080772294

ok16jun120093281991

ok17jul120080772288

ok18jul1020093281985

ok19aug1020083281982

ok20sep1020083281979

ok21dec1020083281976

ok22dec3120100772273

ok23sep3020090772270

ok24apr1020093281968

ok25jun1020100772266

ok26aug120103281964

ok27oct120080772262

ok28oct1020093281960

ok29nov1020083281958

ok30nov120103281956

ok31dec120093281954

ok32jul3120103281952

ok33jan3120103281950

ok34mar3120080772248

ok35feb2820103281946

ok36feb1020083281945

ok37may1020090772244

ok38may120103281943

ok39sep120093281942

ok40sep1020100772241

ok41oct1020103281940

ok42nov1020090772239

ok43feb2920083281938



+ all good dates+ feb last day

Month:jan feb mar apr may jun jul aug sep oct nov decMonth:feb

Day:1 10Day:28

Year:2008 2009 2010Year:2009 2010

Zip code:07722 32819Zip code:07722 32819

+ long month last day+ leap day


Day:31Day:29

Year:2008 2009 2010Year:2008

Zip code:07722 32819Zip code:07722 32819


Month:apr jun sep nov

Day:30

Year:2008 2009 2010

Zip code:07722 32819


MonthDayYearZip codeMonthDayYearZip codeMonthDayYearZip code

jan1200807722jun1200932819nov1201032819

jan10200932819jul1200807722dec1200932819

feb1201032819jul10200932819jul31201032819

mar10201007722aug10200832819jan31201032819

aug31200907722sep10200832819mar31200807722

oct31200832819dec10200832819feb28201032819

nov30200932819dec31201007722feb10200832819

jun30200807722sep30200907722may10200907722

feb28200907722apr10200932819may1201032819

mar1200932819jun10201007722sep1200932819

may10200832819aug1201032819sep10201007722

may31201007722oct1200807722oct10201032819

apr30201032819oct10200932819nov10200907722

feb29200807722nov10200832819feb29200832819

apr1200807722

Testcover.com


Weather Forecast Invalid Date Factor Values

+ too long long month + feb too long, regular yearMonth: jan mar may jul aug oct

decMonth: feb

Day: 32 Day: 29Year: 2008 Year: 2010

Zip code: 07722 Zip code: 07722+ too long short month + feb too long, leap year

Month: apr jun sep nov Month: febDay: 31 Day: 30

Year: 2009 Year: 2008Zip code: 32819 Zip code: 32819

PresenterPresentation NotesThis slide shows the factor values we will give to the test case generator for invalid dates and good zip codes – the second equivalence class. Each box in the slide lists factor values which can be used in a test case; any combination of values in a box gives an invalid date with a valid zip code. Taken together, the 4 boxes specify all the invalid date factor values we want to use in this class.In the first box we have the long months paired with day 32. The second box has short months with day 31. The last two boxes are for February with day 29 and 30.

Weather forecast 2

Testcover.com


Thu Jan 24 21:01:50 EST 2008

Month

Request data:

Month

Day

Year

Zip code

#

+ too long long month


32

2008

07722

+ too long short month

apr jun sep nov

31

2009

32819

+ feb too long, regular year

feb

29

2010

07722

+ feb too long, leap year

feb

30

2008

32819

#1.

TestDayYearZip codeFactor 4Combo

Case IDCountdown


1jan32200807722128

2apr31200932819122

3feb29201007722116

4feb30200832819110

5mar32200807722107

6may32200807722104

7jul32200807722101

8aug3220080772298

9oct3220080772295

10dec3220080772292

11jun3120093281989

12sep3120093281986

13nov3120093281983



+ too long long month+ feb too long, regular year


Day:32Day:29

Year:2008Year:2010

Zip code:07722Zip code:07722

+ too long short month+ feb too long, leap year

Month:apr jun sep novMonth:feb

Day:31Day:30

Year:2009Year:2008



MonthDayYearZip Code

jan32200807722

apr31200932819

feb29201007722

feb30200832819

mar32200807722

may32200807722

jul32200807722

aug32200807722

oct32200807722

dec32200807722

jun31200932819

sep31200932819

nov31200932819

Testcover.com


Weather Forecast Invalid Date Test Cases

Month Day Year Zip Codejan 32 2008 07722apr 31 2009 32819feb 29 2010 07722feb 30 2008 32819mar 32 2008 07722may 32 2008 07722jul 32 2008 07722aug 32 2008 07722oct 32 2008 07722dec 32 2008 07722jun 31 2009 32819sep 31 2009 32819nov 31 2009 32819

PresenterPresentation NotesHere are the test cases for the invalid dates. You can see that January is paired with day 32, with year 2008, and with a good zip code. All the dates are invalid, and the zip codes are valid. So our expected result is a date error message.Now if we had simply mixed the valid and invalid date values together, we might have generated the third test case here – February 29, 2010 with zip code 07722. This test case covers the pair February with day 29 while processing an error because 2010 is not a leap year. Pairwise testing only insures that the February 29th pair will be included once. If the valid and invalid dates are mixed together, February 29th might not be used again to generate an actual weather forecast for the leap day. That pair could be masked and not tested in normal operation.The point here is that different expected results need to be analyzed separately, so that we really test what we intend to.

Weather forecast 2

Testcover.com


Thu Jan 24 21:01:50 EST 2008

Month

Request data:

Month

Day

Year

Zip code

#

+ too long long month


32

2008

07722

+ too long short month

apr jun sep nov

31

2009

32819

+ feb too long, regular year

feb

29

2010

07722

+ feb too long, leap year

feb

30

2008

32819

#1.

TestDayYearZip codeFactor 4Combo

Case IDCountdown


1jan32200807722128

2apr31200932819122

3feb29201007722116

4feb30200832819110

5mar32200807722107

6may32200807722104

7jul32200807722101

8aug3220080772298

9oct3220080772295

10dec3220080772292

11jun3120093281989

12sep3120093281986

13nov3120093281983



+ too long long month+ feb too long, regular year


Day:32Day:29

Year:2008Year:2010


+ too long short month+ feb too long, leap year

Month:apr jun sep novMonth:feb

Day:31Day:30

Year:2009Year:2008



MonthDayYearZip Code

jan32200807722

apr31200932819

feb29201007722

feb30200832819

mar32200807722

may32200807722

jul32200807722

aug32200807722

oct32200807722

dec32200807722

jun31200932819

sep31200932819

nov31200932819

Testcover.com


Weather Forecast Invalid Zip Code

Test casesFactor values+ bad zip code Month Day Year Zip code

Month: feb may feb 1 2009 00001Day: 1 10 may 10 2010 00001Year: 2009 2010 may 10 2009 99999Zip code: 00001 99999 may 1 2010 99999

feb 10 2010 99999

PresenterPresentation NotesThis slide finishes the example with invalid zip codes – the third equivalence class. The box on the left shows the factor values we’ll give to the test case generator. Any combination of values in the box gives a valid date with an invalid zip code. The box on the right gives the test cases for the bad zip codes. All the dates are valid, and the zip codes are invalid. So our expected result is a zip code error message. If we had mixed the bad zip codes into our valid input (the first equivalence class) we could have masked pairs similar to those with the invalid dates. Now I’m not saying we have to do pairwise testing on error messages. It can be done if needed. Again, the point here is that different expected results need to be analyzed separately, so that we really test what we intend to.

Weather forecast 3 input & outp

+ bad zip codeMonthDayYearZip code

Month:feb mayfeb1200900001

Day:1 10may10201000001

Year:2009 2010may10200999999

Zip code:00001 99999may1201099999

feb10201099999

Testcover.com


Pairwise Testing TodayTest demands continue to grow•Pairwise testing is a popular response •for effective coverage with a limited number of test casesImproved training is needed, e.g. to •handle constraints, equivalence classesCareless use has led to controversy•

PresenterPresentation NotesI’d like to turn now to the status of pairwise testing today.Over time we’ve seen expectations continually raised for software testers to increase capacity and accuracy. As computers touch more parts of our lives, there is more to test, and the stakes for "getting it right" continue to go up. Pairwise testing is a very popular response to this demand, by offering effective test coverage with a limited number of test cases. However, in my opinion, improved training is needed, and careless use has led to some controversy.

Testcover.com


Available Tools 1. CATS (Constrained Array Test System)

*) [Sherwood] Bell Labs.

2. OATS (Orthogonal Array Test System) *) [Phadke] AT&T

3. AETG Telecordia Web-based, commercial

4. IPO (PairTest) *) [Tai/Lei]

5. TConfig [Williams] Java-applet

6. TCG (Test Case Generator) *) NASA

7. AllPairs Satisfice Perl script, free, GPL

8. Pro-Test SigmaZone GUI, commercial

9. CTS (Combinatorial Test Services) IBM Free for non-commercial use

10. Jenny [Jenkins] Command-line, free, public-domain

11. ReduceArray2 STSC, U.S. Air Force Spreadsheet-based, free

12. TestCover Testcover.com Web-based, commercial

Pairwise.org Tools List – part 1

PresenterPresentation NotesHow popular has pairwise testing become?One indicator is the list of tools at Pairwise.org. Jacek Czerwonka maintains this web site devoted to pairwise testing. It contains papers and other information about the technique. Here’s the first half of the tools list.____________http://www.pairwise.orghttp://www.pairwise.org/tools.asp

Available Tools

1.

CATS (Constrained Array Test System) *)

[Sherwood] Bell Labs.

2.

OATS (Orthogonal Array Test System) *)

[Phadke] AT&T

3.

AETG

Telecordia

Web-based, commercial

4.

IPO (PairTest) *)

[Tai/Lei]

5.

TConfig

[Williams]

Java-applet

6.

TCG (Test Case Generator) *)

NASA

7.

AllPairs

Satisfice

Perl script, free, GPL

8.

Pro-Test

SigmaZone

GUI, commercial

9.

CTS (Combinatorial Test Services)

IBM

Free for non-commercial use

10.

Jenny

[Jenkins]

Command-line, free, public-domain

11.

ReduceArray2

STSC, U.S. Air Force

Spreadsheet-based, free

12.

TestCover

Testcover.com

Web-based, commercial

Testcover.com


13. DDA *) [Colburn/Cohen/Turban]

14. Test Vector Generator GUI, free

15. OA1 k sharp technology

16. CTE-XL Daimler Chrystler GUI, free

17. AllPairs [McDowell] Command-line, free

18. Intelligent Test Case Handler (replaces CTS) IBM


19. CaseMaker Díaz & Hilterscheid GUI, commercial

20. PICT Microsoft Command-line, free

21. rdExpert Phadke Associates, Inc.

22. OATSGen *) Motorola

23. SmartTest Smartware Technologies Inc. GUI, commercial

24. EXACT *) [Yan/Zhang]

25. AllPairs MetaCommunications Free new

*) Not known to be available publicly

Pairwise.org Tools List – part 2

PresenterPresentation NotesAnd here’s the second half. In all there are more than two dozen tools.To me this indicates a strong need, and a healthy interest in achieving the quality and cost benefits of pairwise testing.However, this enthusiasm has sometimes led to a view that pairwise testing is a silver bullet to be fired without aiming, or a magic spell that needs no understanding of how it works. I have seen articles published for software engineers that support this superficial view. A recent article on orthogonal arrays didn’t address the possibility that real world constraints can lead to impossible test cases. We knew about that 18 years ago. Another paper on pairwise testing had an example that indiscriminately mixed normal and error input values together. I wrote a paper for a STAR conference in 1994 that talked about masking pairs; Czerwonka mentioned it in his 2006 paper for the Pacific Northwest Software Quality Conference. These are not new ideas.____________http://testcover.com/pub/background/star1994_slides.htmhttp://testcover.com/pub/background/star1994_paper.htmhttp://www.pnsqc.org/proceedings/pnsqc2006.pdf

13.

DDA *)

[Colburn/Cohen/Turban]

14.

Test Vector Generator

GUI, free

15.

OA1

k sharp technology

16.

CTE-XL

Daimler Chrystler

GUI, free

17.

AllPairs

[McDowell]

Command-line, free

18.

Intelligent Test Case Handler (replaces CTS)

IBM


19.

CaseMaker

Díaz & Hilterscheid

GUI, commercial

20.

PICT

Microsoft

Command-line, free

21.

rdExpert

Phadke Associates, Inc.

22.

OATSGen *)

Motorola

23.

SmartTest

Smartware Technologies Inc.

GUI, commercial

24.

EXACT *)

[Yan/Zhang]

25.

AllPairs

MetaCommunications

Free

new

*) Not known to be available publicly

Testcover.com


Bach & Schroeder• “Pairwise Testing: A Best Practice that Isn’t”

Useful criticism of careless pairwise testing•MS Word options example • – 31212

Pairs coveredTest cases Pairwise Random

10 100% 88%23 100%

PresenterPresentation NotesPerhaps the best statement of this situation is the 2004 paper by Bach and Schroeder, “Pairwise Testing: A Best Practice that Isn’t.” Here the authors describe several ways testers can use pairwise testing inappropriately. Many of their points can be made about testing in general. They argue that testers should not blindly follow best practices handed down to us, that we need to analyze and take responsibility for our work. In my opinion the paper is on target and worth reading. I recommend it to anyone serious about pairwise testing.However, my interpretation of the paper may differ from others’ on a couple of points. First there is a comparison of pairwise test cases with a set generated randomly. In this example the pairwise test cases cover 100% of the required pairs. The authors observe that an equal number of random tests cover 88% of the pairs; therefore the pairwise tests aren’t much better. But in the same example it takes more than twice as many random test cases to cover all the pairs. If we’re trying to generate a small set of covering test cases, there is no real choice here.____________http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf

Sheet1

Pairs covered

Test casesPairwiseRandom

10100%88%

23100%

Sheet2

Sheet3

Testcover.com


Pairwise Testing Efficiency• Pairwise generators are more efficient than

random test case selection• Significant improvements in pairwise

efficiency are possible• Practitioners need pairwise testing to meet

modern testing challenges

PresenterPresentation NotesIn all the comparisons I’ve seen between pairwise and random generators, the pairwise tools are more efficient. That is, they produce smaller sets of covering test cases.In some of the comparisons, there also seems to be an implicit assumption that the pairwise tests are representative of the technique generally. We just saw a list of 25 tools. The same web page presents a table showing differences in the numbers of test cases they generate. There is variation here, and perhaps more importantly, there are possible improvements, which I’ll talk about shortly.When the time comes to test, we still need combinatorial test tools. As I see it, Bach and Schroeder’s criticisms are not arguments against pairwise testing. They are arguments for doing it well.

Testcover.com


How do Searches Work?

Uncovered pairs

Factor values

?

Test cases

• Go to the space of all possible test cases• Look for small number of test cases (short path)

to minimize the uncovered pairs

PresenterPresentation NotesI’d like to take a few minutes to talk about how pairwise tools search for test cases. All practical test case generators that I’m familiar with do some sort of computer search. A search is necessary to accommodate complex constraints. A search also can be sufficient for generating test cases under broad conditions, using a single algorithm. So it’s a valuable technique that has been widely used.This slide illustrates a test case generator doing a search. It’s something like exploring an alien world. It’s a strange place because each of the test factors represents a different dimension in the space of all possible test cases. It’s also strange because if there are constraints, there are some places we just can’t go. The illustration simplifies things to show the possible test cases as points in the horizontal plane. We can imagine a path from point to point as a sequence of test cases. Before the first test case, all of the pairs we want to test are uncovered. As we select test cases, the number of uncovered pairs is reduced. The search is to find a short path to minimize the number of uncovered pairs. In the illustration, we have a possible test sequence with 4 test cases, and we’re searching for the fifth one. When that next step is taken, the landscape will change, because the number of uncovered pairs for each possible test case depends on which ones have already been covered.

Testcover.com


Search VariationsSeveral algorithms for different tradeoffs•Test practitioner goals:•

Few test cases•Quick responses•Practicality • – mixed values, constraints

Researcher goals:•Fewer test cases•More factors•New methods, better solutions•

PresenterPresentation NotesDifferent search tools use variations on this idea – There is no single pairwise algorithm. Some tools look ahead different amounts; some have restrictions on reconsidering previous test cases; and some make random choices when alternate paths seem equivalent. These design decisions influence how generator tools work and the results they provide. For example, Jacek Czerwonka listed speed of test generation as one of the design principles for Microsoft’s test case generator. In contrast, Kari Nurmela, a researcher at the Helsinki University of Technology had a different goal. He developed a search to find covering arrays with fewer test cases than the best known arrays. That goal takes more processing time.____________http://www.pnsqc.org/proceedings/pnsqc2006.pdfhttp://dx.doi.org/doi:10.1016/S0166-218X(03)00291-9

Testcover.com


Finding the Minimal Set of Test Cases is Hard

?

• Mixed covering array constructions can complement searches using less processing

• Typically searches are CPU bound

• Improvement diminishes with processing time

PresenterPresentation NotesWe want fewer test cases to lower the cost of testing. But finding the smallest possible set of test cases is a hard problem to solve in general. If we do a search, we might find the answer for a small test problem, but searching for the minimal set of test cases becomes impractical as the number of factors and their values increase. So we typically look for answers that are “good enough,” rather than the absolute best.However there are constructions for mixed covering arrays which require less processing, and can provide fewer test cases, compared with a search. I’d like to show a few examples to illustrate the tradeoffs.____________http://dx.doi.org/doi:10.1016/j.disc.2007.07.050

Testcover.com


Change Mixed CA Values Without Searching

Example:•Change OA(9;34) to CA(12;413122)

Reduce the number of values in two of •the factorsIncrease the number of values in one •factor

PresenterPresentation NotesThere are well known ways to transform an array we have to an array we need. To illustrate, let’s suppose we need a mixed covering array that has 1 factor with 4 values, 1 factor with 3 values, and 2 factors with 2 values. It’s very easy get such an array from the orthogonal array in our first example.We’ll do this in two steps. First we’ll reduce the number of values in two factors. Second, we’ll increase the number of values for one of the remaining factors.

Testcover.com


OA(9;34) to CA(9;3222) • Replace all the 2s in any 2 factors with

unassigned (*) values0 0 0 00 1 1 10 2 * *1 0 1 *1 1 * 01 2 0 12 0 * 12 1 0 *2 2 1 0

0 0 0 00 1 1 10 2 2 21 0 1 21 1 2 01 2 0 12 0 2 12 1 0 22 2 1 0

PresenterPresentation NotesHere’s the first step. We’re starting with the orthogonal array having 4 factors of 3 values each. First we pick 2 factors where we want to reduce the number of values from 3 to 2. Let’s take the 3rd and 4th factors. If we keep the values 0 and 1, we don’t need the value 2 any more. So we just replace it with a star, which means we don’t care whether that value is 0 or 1. The array we started with had all the required pairs for 0 and 1 in the 3rd and 4th factors. Those pairs remain, so we still have a covering array.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)



1120OS XDial-upNetscapeApp 2


OS XDSLFirefoxApp 1


XPLANFirefoxApp 2



OS XDSLInt ExplorerApp 2


XPLANNetscapeApp 1

4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

Sheet2

Sheet3

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)





OS XLANFirefoxApp 1








4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


CA(9;3222) to CA(12;413122)Add • 3 rows with 3s to one of the factors with 3 valuesFor each of the other •factors, pair its values with the 3s in the new rows

0 0 0 00 1 1 10 2 * *1 0 1 *1 1 * 01 2 0 12 0 * 12 1 0 *2 2 1 03 0 0 03 1 1 13 2 * *

PresenterPresentation NotesNow for the second step, we take the array from the previous slide (it’s here in the first 9 rows) and increase the number of values for one factor. Let’s pick the first factor. We add 3 rows to the bottom and put the next value (3) in the first column. All the pairs for 0, 1 & 2 in the first column are already contained in the original array. So the only new pairs we need are associated with the new value. To include them, all we have to do is put the allowed values for the other factors in their respective columns. The covering array we needed is the result.So we see that there are simple ways to change the numbers of values without doing much processing. A tool that takes advantage of these transformations can benefit. What if we need to change the number of factors in an array? There’s good news here as well.

tables

+012x012

00120000

11201012

22012021

3 factors

+012x012

00120000

1120111

2201222

1012

120

201

2021

102

210

4 factors

+012

00120000

11200111

22010222

1012

1120

1201

2021

2102

2210

test def





test conf






OS XDSLFirefoxApp 1





test conf (2)






OS XDSLFirefoxApp 1





test conf (3)



1120OS XDial-upNetscapeApp 2


OS XDSLFirefoxApp 1


XPLANFirefoxApp 2



OS XDSLInt ExplorerApp 2


XPLANNetscapeApp 1

4 factors (2)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

4 factors (3)

+012

00120000

11200111

220102**

101*

11*0

1201

20*1

210*

2210

3000

3111

32**

Sheet2

Sheet3

Testcover.com


More CA Factors from Products and Searches

Product sequence:•OA(• 9;34), CA(15;315), CA(21;357), …

Nurmela• ’s tabu search:CA(• 15;320)

Products using tabu search result:•CA(• 15;320), CA(21;374), CA(27;3282), …

Combined methods give better results•

PresenterPresentation NotesIt’s possible to increase the number of factors in a covering array using a product construction. We add some rows (a few more test cases) to a covering array, and that allows us to multiply the number of factors. For example, starting with the same orthogonal array with 4 factors, we can add 6 rows and get almost 4 times as many factors. And again, 6 more rows give us almost 4 times as many factors. This can be repeated indefinitely, without a great deal of processing.Kari Nurmela’s search that I mentioned earlier does improve on the array above. Using 15 test cases it covers 5 more factors. But the interesting thing here is that products using Nurmela’s array are even better than using searches or products alone.____________http://dx.doi.org/doi:10.1002/jcd.20065http://dx.doi.org/doi:10.1016/S0166-218X(03)00291-9

Testcover.com


Optimal Mixed CAs & More Factors• Add n test cases to OA(25;56) to get optimal array• Each factor of 5 values can be replaced by the

indicated factors without additional test cases

Example: Replace • 3 factors in CA(35;7155) to get CA(35;715248360)

n Optimal Array Replacement Factors0 OA(25;56) 35 26,435

5 CA(30;6155) 312 287,516

10 CA(35;7155) 44 360 21,410,864

15 CA(40;8155) 44 390 219,969,152

20 CA(45;9155) 55 416 3324 2320,932,800

25 CA(50;10155) 55 416 3810 24,653,525,600

PresenterPresentation NotesI saved my favorite example for last. I found this column expansion construction while working on the Testcover.com service. It enables us to increase the numbers of factors in mixed covering arrays without doing a search. Even better, the arrays are optimal. That means we have the minimum number of test cases for these factors and their values.The idea here is to replace an individual factor in an array with more factors having fewer values. Let’s suppose we need a mixed CA(35;715248360). We start with an OA(25;56) and add 10 rows to increase the number of values to 7 for 1 factor. This covering array has 5 factors of 5 values. We replace 2 of these factors with 4 factors of 4 values each; that’s 8 factors in all. Then we replace another factor of 5 values with 60 factors of 3 values, and we have the array we need.____________http://dx.doi.org/doi:10.1016/j.disc.2007.11.021

Sheet1

nOptimal ArrayReplacement Factors

0OA(25;56)3526,435

5CA(30;6155)312287,516

10CA(35;7155)4436021,410,864

15CA(40;8155)44390219,969,152

20CA(45;9155)5541633242320,932,800

25CA(50;10155)55416381024,653,525,600

Sheet2

Sheet3

Testcover.com


Mixed Covering ArraysCompared with OA templates•

More choices for numbers of values•Fewer test cases typically•

Compared with greedy search•Faster responses (less processing)•Fewer test cases when CAs are optimal•

Search needed for constraints•

PresenterPresentation NotesIn summary, the properties of mixed covering arrays offer more flexibility compared with orthogonal arrays alone. There are more choices for array parameters, and covering arrays often result in fewer test cases.Constructing a mixed covering array can require less processing than a search and may result in a faster response time. When an optimal covering array is available, no amount of searching will improve on the number of its test cases.On the other hand, practical test case generators still need a search capability to handle complex constraints.

Testcover.com


Future Pairwise Trends• Platforms:

• Faster, distributed processing• Tool Software:

• Smarter searches, less processing• Fewer test cases

• Improved test efficiency• Faster responses

• What-if test designs, test changes• Use with agile, incremental cycles

PresenterPresentation NotesWhat does this mean for future pairwise testing?One of the things I find amusing about the old CATS t

pairwise testing comes of age - testcover.comtestcover.com/pub/background/stareast2008s.pdfthis talk...

Documents