constraint-based modeling of interoperability problems using an object-oriented approach

Constraint-Based Modeling of InterOperability Problems Using an Object-Oriented Approach

Mohammed H. Sqalli

Eugene C. Freuder

University of New Hampshire

msqalli,[email protected]

-The first author is currently at the of Canada Ltd.

-This material is based in part on work supported by the National Science Foundation.

IAAI-01: Sqalli & Freuder 2August 7, 2001

Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


Outline

ExampleExample Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


InterOperability Testing

Specifications (What should Happen)

Observations (What really happens)

Communication line

InterOperability Testing <=> Compare (Specifications, Observations)

Device BDevice A


Example from a Test Suite

Test Case ID: V4202H__004 Test Purpose: Verify that the first Hello sent from

both sides contains Remote node ID and Remote port ID set to zero (1Way).

Pre-requisite: Both SUTs (A, B) are in different lowest level peer groups (Out).

Verdict Criteria: The first Hello packet observed from each SUT will have the Remote node ID field and Remote port ID field set to zero (1WayOut).


Example - Test Description

Switch A Switch B

1WayOut

1WayOut


Example - CSP Model

Node IDPort ID

Remote Node ID

Status

Source

1WayOut(A)

Variables

Unary

Binary

Constraints

1WayOut(A) {Source, Time, Status, Node_ID, ...}

1WayOut(B) {Source, Time, Status, Node_ID, ...} ...

1WayOut(A).Status = Mandatory

1WayOut(A).Remote_Node_ID != 0

1WayOut(A).Time < 1WayOut(B).Time

1WayOut(A).Peer_Group_ID != 1WayOut(B).Peer_Group_ID ...

1

0

Remote Port ID

Type

Peer Group ID

Time

Hello

Node IDPort ID

Remote Node ID

Status

Source

1WayOut(B)

1

0

Remote Port ID

Type

Peer Group ID

Time

Hello

<

!=

0 0

!=


ADIOP- Test Suite Builder GUI

Panel showing the CSP model of a test case

Panel for declaring variables and domains

Panel for declaring constraints


ADIOP - Test Suite Menu GUI


Protocol Specification

Monitored Observations

Result

Decoded Observations

Test SuitesTest #1

Test #2…

CSP ModelsCSP #1CSP #2

…

CBR + ExpertResult

Report

OK

NoYes

?StoreNext Step

Debugging

ModelingDiagnosis

Decoding

ADIOP – The big picture (Four Modules)

Update


ADIOP - Modeling


Test SuitesTest #1

Test #2…


…Modeling


Outline

Example Domain of ApplicationDomain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


InterOperability Lab

Verify the interoperability and/or conformance of the computer communications products

Consortiums: ATM, FDDI, Wireless, … Protocols tested: UNI, LANE, PNNI, … Test Suites for IOP testing: LANE, PNNI, … Testing Equipment: Snifters, Generators,... Testing Problems: Data Analysis, Similar

Problems, ...


ATM and PNNI

Asynchronous Transfer Mode– supports all classes of traffic: voice, video, and data– provides quick and easy switching and multiplexing– connection oriented

Private Network Network Interface– provides dynamic routing: includes discovery of the

topology of the network– supports QoS and hierarchical routing– scales to very large networks


InterOperability Testing

NetworkMonitorDevice A Device B

Protocol SpecificationsCompare to

Report

Observations


Outline

Example Domain of Application Motivations and ContributionsMotivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


Motivations – The ADIOP System

Diagnose IOP problems in a timely manner Analyze a large amount of data Keep trace of previous problems and their

solutions to be reused in future similar cases Debug IOP tests


Solution - The ADIOP System

Automate the process of InterOperability Testing– CSP modeling of IOP tests– Diagnosing IOP problems quickly and efficiently

using CSP models (including Explanation) Debug the InterOperability tests


Contributions

Model building: Simple, declarative, transparent, GUI-based, and complete modeling language for stating and generating IOP tests.

OO provides a more natural, concise, scalable and reusable framework for model statement of IOP tests.

Combination of CSP and OO: metavariables and metavalues to represent packets and generate test cases as objects.

Modeling is generic across protocols: independent of the protocol being used.

Scalability: More protocols can be made available for modeling by adding the corresponding decoders.

Dynamic loading of protocols and packets being used. Many-models architecture: each test case is represented by a CSP model. Constraint expressiveness: unary and binary constraints.


Outline

Example Domain of Application Motivations and Contributions CSP ModelingCSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


Constraint Satisfaction Problems

Values

Constraints

Problem Statement

Variables

CSP Algorithm Solution

X

ZY

Coloring Problem

CSP Representation

• Search

•Inference

X: red

Y: blue

Z: green

Redgreenblue

Redgreenblue

Redgreenblue

Z

X

Y

Z

X

Y


One Model vs. Many Models

One Model (Protocol Specification) [Sqalli & Freuder 1996]– Only one model to use

for testing, but too complex

– Less inconsistencies– No redundant testing

Many Models (Test Suite) [Sqalli & Freuder 1998]– Easy and small models for

specific tests To state To fix/update

– Suitable for interoperability testing

Generate reports Preferred by vendors/testers

– Explanation of the cause of failure


CSP Modeling of InterOperability Testing



Result


Test SuitesTest #1

Test #2…


…

NB: The decoders have built in functionality that provides the information (protocols, packets, packet’s fields) to build the CSP models.


Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with ObjectsModeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation Summary


Modeling with Objects

Packet ...Time Source Type Node ID

Object(Metavariable/

Metavalue)

•Time

•Source

•Type•Node ID

getNodeID

getTime

getPacket

CSP

Metavariable: X = {x1, x2, …, xn}, where xi is a variable

Metavalue: V = {v1, v2, …, vn}, where vi is a value

Binary Metaconstraint: Cxy= {c1, c2, …, cm}, where ck is a constraint between xi and yj


Class Hierarchy

Packet Time Source Info

PNNI Routing Packet

Time Source PNNI Routing Header Info

Hello Packet Time Source Node ID ...PNNI Routing Header

Packet

Mpoa PnniRout Lane

Hello Dbs


Advantages of Object-Orientation

Inheritance between objects allows for a hierarchical definition of packets which matches the way protocols are specified.

Encapsulation – Information hiding of the objects definition as the users do not need to

know the details but only the functionality.– Modularity: Each object is a separate entity with its own functionality.

Scalability: Adding more protocols and packets does not affect the already existing system.

Reusability: The same objects are being used for decoding packets as well as for CSP model definition.

Natural: The CSP model obtained using objects is concise and expressive.


Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and LanguageModeling Interface and Language Test Cases as Objects Application & Evaluation Summary


Modeling Interface (GUI)

Protocols List: loaded from the class hierarchy

of the Packet class

Packet Types List: loaded from

the class hierarchy of the protocol’s class

Packet’s Parameters (CSP variables) List: loaded

from the class definition of a packet.


Advantages of CSP Modeling using OOP

No need for the tester to know what each packet contains (built in the hierarchy).

No need to know the syntax of the modeling language (generated by the GUI)

Possibility to add more protocols just by adding the decoder for such protocol (protocol independent modeling).

Dynamic loading of information about protocols, packets, etc. (done at runtime).


Example from a Test Suite

Test Case ID: V4202H__004 Test Purpose: Verify that the first Hello sent from

both sides contains Remote node ID and Remote port ID set to zero (1Way).

Pre-requisite: Both SUTs (A, B) are in different lowest level peer groups (Out).

Verdict Criteria: The first Hello packet observed from each SUT will have the Remote node ID field and Remote port ID field set to zero (1WayOut).


CSP Model

Node IDPort ID

Remote Node ID

Status

Source

1WayOut(A)

Variables

Unary

Binary

Constraints

1WayOut(A) {Source, Time, Status, Node_ID, ...}

1WayOut(B) {Source, Time, Status, Node_ID, ...} ...

1WayOut(A).Status = Mandatory

1WayOut(A).Remote_Node_ID != 0

1WayOut(A).Time < 1WayOut(B).Time

1WayOut(A).Peer_Group_ID != 1WayOut(B).Peer_Group_ID ...

1

0

Remote Port ID

Type

Peer Group ID

Time

Hello

Node IDPort ID

Remote Node ID

Status

Source

1WayOut(B)

1

0

Remote Port ID

Type

Peer Group ID

Time

Hello

<

!=

0 0

!=

Metavariable Metavariable


Test Suite Builder GUI

Step 5: This generates the test case object

Step 1: Choose the protocol

Step 2: Start CSP Model

Step 3: Define the packets

Step 4: Define the constraints


Example of CSP Modeling- Modeling Language -

$CSP

$PROTOCOL PnniRout

$PACKET OneWayOutA Hello

$PACKET OneWayOutB Hello

$UNARY_CONSTRAINT OneWayOutA.status D_Mandatory

$UNARY_CONSTRAINT OneWayOutA.remote_port_id == 0

$UNARY_CONSTRAINT OneWayOutB.remote_port_id == 0

$UNARY_CONSTRAINT OneWayOutA.remote_node_id == 0

$UNARY_CONSTRAINT OneWayOutB.remote_node_id == 0

$BINARY_CONSTRAINT OneWayOutA.time < OneWayOutB.time

$BINARY_CONSTRAINT OneWayOutA.source != OneWayOutB.source

$BINARY_CONSTRAINT OneWayOutA.peer_group_id != OneWayOutB.peer_group_id

$ENDCSP

MetavariablesOptional statemen

t

Unary Constraint

s

Binary Constraint

s

Protocol


Modeling Language Definition

$CSP $ENDCSP $PROTOCOL protocolTested $PACKET packet_name packet_type $DOMAIN domain value1 value2 … $UNRAY_CONSTRAINT variable operation [domain|value] $BINARY_CONSTRAINT variable1 operation variable2 $CONSTRAINT variable1 variable2 f(variable1, variable2)


Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as ObjectsTest Cases as Objects Application & Evaluation Summary


This will generate an object in which fields are domains or metavariables, and methods are constraints. (This is the input to the Diagnosis module)

Example of CSP Modeling- Test Cases as Objects -

OneWayOutA

OneWayOutB

HelloOneWayOutA_Status

OneWayOutB_Status

OneWayOutA_Source_OneWayOutB_Source

•Mandatory

•HelloOneWayOutA_Type

OneWayOutB_Type

OneWayOutA_Time_OneWayOutB_Time


Test Suite Menu (GUI)


Test Suite Directory Hierarchy

testsuite

mpoa pnnirout lane

V4301H_001

adiop

adiopx

V4301H_002 V4301H_003

Test cases are stored in the appropriate directory. Menu of test cases is generated dynamically from this directory structure.


Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & EvaluationApplication & Evaluation Summary




Result


Test SuitesTest #1

Test #2…


…

CBR + ExpertResult

Report

OK

NoYes

?StoreNext Step

Debugging

ModelingDiagnosis

Decoding

ADIOP – The big picture (Four Modules)

Update


ADIOP(Automated Diagnosis of

InterOperability Problems)

Test Suite Builder

Test Generation and Implementation

Test Debugger

Diagnoser

Test Execution(Algorithms)

Menus and Reports

Decoder s Builder Decoder Usage

Decoder

Search(hard explanation)

Inference(better explanation)

Automate Menus Creation

Reports Generation

CBR + User Interaction

Correct/Update CSP Model

New CSP Model


Frame Format(Defined language)

Java file(Frame decoder)

Java Class(Frame decoder)

CSP Model

Notes:

Implemented

To be implemented

Not implemented

Existent Documents

Test Description from IOP test suite

Natural Language Processing

Graphical Representation

CSP Model

Java file

Java Class(used by diagnoser)

Packet

PnniR MPOA

Hello DBS

Note: These classes are used by Decoder, Diagnoser, and Test Builder.

Hierarchy of Classes


Application of CSP Modeling - ADIOP Diagnosis

The test cases built using the ADIOP’s modeling component are dynamically accessible through a menu.

Motivations of Diagnosis of IOP Problems: save time, reduce repetitive testing, store and reuse knowledge, automate reports generation.

Advantages of using CSP for modeling and diagnosis: take advantage of existing CSP algorithms.

Algorithms for Diagnosis: Search and Inference. Generate an accurate and human-like explanation. Evaluation and summary numbers: 50 (73% ) test cases out of

69 built produced a meaningful explanation.


Outline

Example Domain of Application Motivations and Contributions CSP Modeling Modeling with Objects Modeling Interface and Language Test Cases as Objects Application & Evaluation SummarySummary


Summary

CSP Model building: Simple, declarative, transparent, GUI-based, and complete modeling language for stating and generating IOP tests.

OO provides a more natural, concise, scalable and reusable framework for model statement of IOP tests (+ Inheritance, Encapsulation)

Combination of CSP and OO: metavariables and metavalues to represent packets and generate test cases as objects.

Scalability: More protocols can be made available for modeling by adding the corresponding decoders.

Modeling Interface: dynamic loading of protocols and packets being used. Modeling Language:

– Generic across protocols: Independent of the protocol being used.– Constraint expressiveness: unary and binary constraints.

Many-models architecture: test suites are used instead of protocol specifications. Each test case is represented by a CSP model.

Application & Evaluation: Diagnosis and Explanation of IOP problems.

Constraint-Based Modeling of InterOperability Problems Using an Object-Oriented Approach

Discussion

Mohammed H. Sqalli

Eugene C. Freuder

University of New Hampshire

msqalli,[email protected]

constraint-based modeling of interoperability problems using an object-oriented approach

Documents

sqalli freudermotivations

sqalli freuderatm

language test cases

test suites

test purpose

testing problems

objectsmodeling interface

test suitetest case