1

Common Patterns in Combinatorial Models

Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick

IBM Haifa Research Labs

2

Agenda

• Introduction & Motivation

• Pitfalls in Combinatorial Models

• Modeling Patterns

3

Introduction

• Combinatorial model = attributes, values, restrictions

• Correctly identifying them is of the main obstacles in practicing combinatorial testing

• There are common pitfalls, and common patterns that address them

4

Motivation

• Construct a “checklist” for practitioners– Common pitfalls and common solutions

• Raise the topic for community discussion– This is not a complete list– More patterns are welcome

5

Common Pitfalls in Combinatorial Modeling

Correctness

Failing to capture the intention correctly

Completeness

Omitting an important part of the test space from the model

Redundancy

Explicitly enumerating different cases that are actually equivalent

6

Modeling Patterns

• Recurring patterns we encountered in combinatorial models– i.e., recurring properties of the modeled test

spaces

• Address the pitfalls

• Encountered in many different models, regardless of the domain, current level of testing, etc.

7

Overview

8

Optional and Conditionally-Excluded Values (completeness, correctness)

• Example I: online form with fields email (mandatory) and home address (optional)

• Naïve model:– Email – valid / invalid– HomeAddr – valid / invalid

• Much better:– Email – valid / invalid– HomeAddr – valid / empty / invalid

Incomplete !

Does not distinguish between empty and

invalid home address

9

Optional and Conditionally-Excluded Values – cont.• Example II: online form with email

address, home address, and “is billing by email”

• Home address used only if not billing by email

10

• Naïve model: – Email – valid / invalid– HomeAddress – valid / invalid– BillByEmail – true / false

– Not allowed:• HomeAddress == “valid” && BillByEmail• HomeAddress == “invalid” && BillByEmail

Incorrect !

Billing by email was entirely excluded

Optional and Conditionally-Excluded Values – cont.

11

Optional and Conditionally-Excluded Values – cont.

• Much Better:– Email – valid / invalid– HomeAddr – valid / invalid / empty / NA– BillByEmail – true / false

– Not allowed:• HomeAddr != “NA” && BillByEmail• HomeAddr == “NA” && ! BillByEmail

12

Multi-Selection (correctness, completeness)

• Example: shopping cart system. The shopping cart may contain meat, dairy, fish and drinks

• Naïve model:– Cart Item – meat / dairy / fish / drinks

Incomplete and Incorrect !

Does not allow for the cart to contain more

than one item, and does not test interactions

between items

13

Multi-Selection – cont.

• A better solution:– Item 1 – meat / dairy / fish / drinks

…– Item t – meat / dairy / fish / drinks / none

• An even better solution:– Meat – true / false– Dairy – true / false– Fish – true / false– Drinks – true / false

14

Ranges and Boundaries (redundancy, completeness)

• Applied when values of a parameter can be divided into ranges, such that values in a range are equivalent with respect to the testing scenario

15

Ranges and Boundaries – cont.

• Example I: file processing. File of up to 1 MB is handled by a first approach. File of 1-256 MB by a second, and over 256 by a third

• Model:– FileSize: 0 / moreThan0LessThan1 /

1 / moreThan1LessThan256 / 256 / moreThan256

16

Ranges and Boundaries – cont.

• Example II: user updates insurance plan. User chooses benefits for the new plan, some may already exist.

• Model:– Benefits – allBenefitsExisting /

someExistingSomeNew / allBenefitsNew

17

Order and Padding (completeness)

• When? When a requirement defines an output that depends on two (or more) inputs.

• Modeler Q: Can bugs surface if inputs are presented in different orders ?

18

Order and Padding – cont.

• Example: expense reimbursement. Expenses covered only if both food and lodging expenses are submitted

• Naïve model:– Lodging – true / false– Food – true / false

Incomplete !

The following bug may not be detected

19

Order and Padding – cont.

if (record.type == “Food” && pLodgingWasSubmitted){ process(record); }

if (record.type == “Lodgin” && pFoodWasSubmitted){ process(record); }

20

Order and Padding – cont.

• Better model:– Lodging – true / false– Food – true / false– FoodBeforeLodging – true / false /

NotApplicable

21

Order and Padding – cont.

• Modeler Q II: may the implementation depend on the records being consecutive?

• Modeler Q III: may the implementation depend on either records being first / last?

22

Order and Padding – cont.

• Updated model:– Lodging – true / false– Food – true / false– FoodBeforeLodging – true / false /

NotApplicable– PaddingBefore – true / false / NotApplicable– PaddingBetween – true / false / NotApplicable– PaddingAfter – true / false / NotApplicable

23

Multiplicity and Symmetry (redundancy)

• Applicable when:– Multiple elements of the same type– Their interactions with other elements are

equivalent

• Example: two hosts (x86 / PowerPC), two storage devices (Enterprise / Mid-Range / Low-End)

24

Multiplicity and Symmetry – cont.

• Naïve Model:– Host1 – x86 / PowerPC– Host2 – x86 / PowerPC– Storage1 – Enterprise / MidRange / LowEnd– Storage2 – Enterprise / MidRange / LowEnd

Warning: Possible Redundancy !

For example, (host1=x86, Storage1=LowEnd)

may be equivalent to (host2=x86, Storage1=LowEnd)Warning: Possible Redundancy ! For example, (host1=x86, host2=PowerPC)

may be equivalent to (host2=x86, host1=x86)

25

Multiplicity and Symmetry – cont.

• Possible Solution:– numX86Running – 0 / 1 / 2– numPowerPCRunning – 0 / 1 / 2– numEnterpriseRunning – 0 / 1 / 2– numMidRangeRunning – 0 / 1 / 2– numLowEndRunning – 0 / 1 / 2– Not allowed:

• numX86Running + numPowerPCRunning != 2• numEnterpriseRunning + numMidRangeRunning +

numLowEndRunning != 2

26

Multiplicity and Symmetry – cont.

• An alternative approach:

• A tool and algorithm that supports designating equivalent parameters– And whether internal interactions are

equivalent

• Requires a tool that supports dynamically changing coverage requirements.

27

Summary

• Modeling is typically the most challenging part in applying combinatorial testing

• We listed some pitfalls, and common patterns that address them

• There are probably many more

28

Thank You For Listening