testing techniques

Testing Techniques

Conformance Testing Methodology

and Framework

ISO IS-9646

TT 2004, ISO Conformance 2© Ed Brinksma/Jan Tretmans

Overview

Conformance Testing Methodology and Framework

ISO IS-9646

Literature: “An Overview of OSI Conformance Testing”


Overview: This Lecture

Protocol conformance testing: why and what Standardization Overview Standard IS-9646:


Conformance: static and dynamic Options, protocol implementation conformance statement Test generation: test purposes, abstract-, executable tests Test methods, test architectures, Test classification and -structuring Test notation Test implementation and -execution


Types of Testing


Protocol Conformance Testing

distributed systems

exchange of information

rules for communication : protocol

standardized protocol specifications

conforming implementations

testing for protocol conformance



International Standard ISO IS-9646 :


How to test protocol entity implementations

for conformance with respect to their specifications

Originally for OSI systems

Now more generally applied

Defines test language TTCN

Standard accepted by ITU-T, ETSI, ATM-Forum, ……



Starting point :a correct and valid (standardized) protocol specification

Original goal :standardization of test suites in TTCN consistency between specification and test suite ?

Conformance testing interoperability testing


The Conference Protocol


CPE

UDP Layer

CPECPE

The Conference Protocol

joinleavesend

receive

Conference Service


Conference ProtocolTest Architecture

CPE= IUT

UT-PCO = C-SAP

LT-PCO

UDP Layer

U-SAP LT-PCO

TesterTorX

B C

A


(N)-ASPs (N)-ASPs

(N)-service provider

(N)-ProtocolEntity

(N)-PDUs

(N-1)-ASPs

(N-1)-service provider

(N)-ProtocolEntity

protocol

(N-1)-ASPs

Protocol Conformance Testingvs. Interoperability Testing


Protocol Conformance Testingvs. Interoperability Testing

Protocol Conformance Testing: correct protocol (design and validation) conforming protocol entities (testing) interoperability

Interoperability testing test directly whether each pair of protocol entities can

work together (i.e. deliver (N)-service)

Internet: interoperability testing

with (standard) reference entities


Standard Conformance Testing

Testing by: manufacturers users organizations telecom operators independent test laboratories

Standard IS-9646 for certification of implementations accreditation of testing processes and test laboratories international acceptance of test results


ISO IS - 9646

Part 1 : Introduction and general concepts

Part 2 : Abstract test suite specification

Part 3 : TTCN

Tree and Tabular Combined Notation

Part 4 : Execution of tests

Part 5 : Requirements on test laboratories and

administrative conformance assessment

process


testgeneration

testexecution

testimplementation

standardprotocol

specification

standardconformance

test suite

verdict

implementationprocess

protocolimplementation

IUT

process ofconformance testing

executabletest suite


Conforming Implementation

Specifications in natural language

What is a conforming implementation ?

“ … exhibits conformance

if it complies with the conformance requirements

of the applicable specification standard …”

Satisfaction of conformance requirements

Conformance requirements :

“the implementation shall do ….. “

“the implementation shall not do …..”

Complication : options



(P)ICS -

(Protocol) Implementation Conformance Statement :

list of implemented options

supplied by manufacturer

(P)ICS proforma

fill-in questionnaire with all possible options

attachment to protocol standard

to be completed by manufacturer to make PICS

Restrictions on combinations of implemented options

documented in standard in static conformance requirements


Example : Static Conformance

Requirement

Feature A is optional

Feature B is optional

But A may only be implemented

if B is also implemented



Requirements on dynamic behaviour: dynamic conformance requirements

Messages : structure coding

Ordering : sequence of sent and receive messages relation between information content of different messages

Example: When in state Enable, upon receiving a Data-Request,

the implementation shall send a Data-PDUor it shall enter the abort phase.



A conforming implementation is onewhich satisfies both static and dynamic conformance requirements, consistent withthe capabilities stated in the PICS

Conformance testing: checking static conformance by reviewing checking dynamic conformance requirements by testing


Test Generation

dynamic conformance requirement

test purpose(s)

generic test case

abstract test case

executable test case

test case execution


Test Generation

Test purpose :precise description (natural language)of what is to be tested in order to decide aboutsatisfaction of a particular conformance requirement

Example: When in state Enable, upon receiving a Data-Request,

the implementation shall send a Data-PDUor it shall enter the abort phase.

Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent.

(but implementation may also enter the abort phase!)


Test Generation

Generic test case:operationalization of test purpose;description of actions to be performedto reach test purpose

Example:

Check whether in state Enable, after receivinga Data-Request, a Data-PDU is sent.

Go to state Enable, send a Data-Request,and observe a Data-PDU, and if not check that theimplementation starts entering the abort phase


Test Generation

Abstract test case:take test method into account

Test method ( test architecture, test environment ) :abstract description of environmentin which testing will be performed,in terms of PCO - Points of Control and Observation

Example:Do Connect-Request,

check Connect-Response (now in state Enable),send Data-Requestobserve Data-PDUor observe Start-Abort


IUT

LT

UT

(N)-ASPs

(N)-PDUs

PCO

PCO(N-1)-ASPs

testcoordination

Local test method

Test Methods


IUT

LT

UT

(N)-ASPs

(N)-PDUs

PCO

PCO (N-1)-ASPs

testcoordination

service provider

test system SUT

Distributed test method

Test Methods


Test Notation

Abstract test cases are written in standardized notation :TTCN : Tree and Tabular Combined Notation

Specification language for Abstract Test Cases

Behaviour specified as tree of sequences of possible events

Data specified in tables or in ASN.1

ISO IS-9646 Part 3

With: input ? , output ! (from tester perspective!) ,timers / time out, ordering of events,default behaviour, alternative behaviours,tree attachment (=subroutines),parallel component testers,

verdicts: pass, fail, inconclusive


behaviour

Test Case Dynamic Behaviour

Test Case Name: Conn_EstabGroup: transport/connectionPurpose: Check connection establishment with remote initiative

constraints verdict

+preamble LT ! T-PDU-connect-request UT ? T-SP-connect-indication UT ! T-SP-connect-response LT ? T-PDU-connect-confirm OTHERWISE LT ? T-PDU-disconnect-request OTHERWISE

passfailinconclusivefail

Example : TTCN


Classification of Tests

Basic interconnection tests

Capability tests

Behaviour tests

Conformance resolution tests


Structuring of Test Suites

Test suite

Test caseTest case Test case

Test stepTest step Test step

Test eventTest event Test event

PCO ? X PCO ! A

Test groupTest group Test group


Test Implementation

Transform abstract test suite into executable test suite by compilation interpretation a test slave …..

Select test cases from abstract test suite based on PICS Incorporate (P)IXIT -

(Protocol) Implementation eXtra Information for Testing : address information timer information specific parameter values …..

PIXIT proforma supplied by test laboratory


Test Execution

Execute / interpret executable test suite

Collect and analyse the resultsbased on verdicts pass, fail, inconclusiveof each test case execution

Collect all results of static conformance review

dynamic testing

into Protocol Conformance Test Report


implementationprocess

staticconformance

review

testexecution

analysisof results

testselection

testimplementation

standardizedtest methods

testpurposes

generictest suite

testnotation

verdicttest report

certification

PIXITproforma

standard protocolspecification

PICSproforma

dynamic conformance req’s

static conformance req’s

protocolimplementation

IUT

PIXIT

PICS

(standardized)abstracttest suite

Basic interconn.capabilitybehaviour

executable

test suite

testing techniques

Documents