overview of etsi testing methodology ict-osa/parlay workshop brazil, march 2006

Overview of ETSI Testing Methodology

ICT-OSA/Parlay WorkshopBrazil, March 2006

Anthony Wiles

Manager

ETSI Protocol and Testing Competence Centre

Testing Methodology - ICT Workshop, Brazil March 2006 2

Interoperability is …

The ultimate aim of ICT standardisation The red thread running through the entire standards

development process, it’s not an isolated issue Not something to be somehow fixed at the end

The ability of systems and products to interwork is fundamental

ETSI approach ETSI produces with the required degree of parallelism

• Base Standards and • Test Specifications

Base standards should be designed with interoperability in mind Profiles designed to reduce potential non-interoperability Protocol conformance and Interoperability statements Two complementary forms of testing

• Conformance testing• Interoperability testing (formal IOT)

Testing provides vital feedback into the standardization work


Technical Committee MTS (Methods for Testing and Specification)

Development of methodologies, techniques and languages (including TTCN3) http://portal.etsi.org

ETSI’s Protocol and Testing Competence Centre (PTCC)Supports ETSI committees on the application of formal techniques in standards on a daily basisDevelopment of test specifications (conformance and interop)

http://www.etsi.org/ptcc

ETSI Plugtests™ ServiceValidation of standards and prototypes through interoperability events http://www.etsi.org/plugtests

These resources are highly complementary

ETSI Resources for Interoperability


Typical Standards Development Process in ETSI

(Unit) Conformance Testing

Interoperability Testing

Products mature from prototypes to commercial products

ETSI: Development of base standards

Certification

Industry

time

Conformance Test Specifications

Interoperability Test SpecificationsIterative feedback


Different Kinds of ETSI Test Specifications

Conformance Robustness

PerformanceInteroperability

Network Integration

RF/EMC


Progressive Testing

All engineering disciplines accept that testing should be applied to progressively complex units From individual components to entire systems

This demands a number of different testing solutions There is no silver bullet

In the two extremes Just testing the components is not enough Just testing the system is not enough

Limitations are economic, not technical What can I afford to test? What can I afford not to test?

What should be covered by standardised test specifications What is the right kind of testing for the job

What kind of ‘thing’ is being tested? Device? System? Service? What is the most economic method(s)? What resources are (or must be made) available?

• Tools, test scripts, testbeds etc.


Conformance and Interoperability Testing are Complementary

ETSI experience As you move up a system stack the emphasis should change

from conformance to IOT

Lower layer protocols, infrastructure Emphasis on conformance

Middleware, enablers Combination of Conformance + IOT

Services, applications, systems Emphasis on IOT

Conformance testing as a pre-requisite to IOT

Interop Testing

Conformance Testing

Interoperability


Conformance Testing

Tester

Is Black-Box testing Stimulation and Response

Device

Point of Control and Observation (PCO)

Reqs.


Conformance Testing Tests is Usually Layer-byLayer

lat

ii

1 2 3

4 5 67 8 9

* 8 #

latigid

TestSystem

SUT API

MMI

L3

L2

PHY

SUT = System Under Test

IUT = Implementation Under Test

L3

Upper Tester

Lower Tester


Characteristics of Conformance Testing (1)

Is unit testing Tests a single ‘part’ of a device (e.g., a protocol layer)

• Often incrementally – layer-by-layer

Tests against well-specified requirements For conformance to the requirements in a base

specification or profile (standard) Usually limited to one requirement per test

Tests at a 'low' level At the protocol (message/behaviour) level (bits)

• Or service primitive, API, Interface

Tests over standardised interfaces May not be available to ‘normal’ user

Requires a test system (and executable test cases) Can be expensive (e.g., radio-based test system) Tests under ideal conditions


Characteristics of Conformance Testing (2)

High control and observability Means we can explicitly test error behaviour Can provoke and test non-normal (but legitimate)

scenarios Can be extended to include robustness tests

It is usually automated and tests are repeatable Conformance Testing is DEEP and NARROW

Thorough and accurate but limited in scope Gives a high-level of confidence that key components of

a device or system are working as they were specified and designed to do


Limitations of Conformance Testing

Does not prove end-to-end functionality (interoperability) between communicating systems Conformance tested implementations may still not interoperate

• This is often a specification problem rather than a testing problem! Need minimum requirements or profiles

Does not test a complete system Tests individual system components, not the whole

• A system is often greater than the sum of its parts!• Does not test functionality

– Does not test the user’s ‘perception’ of the system

Standardised conformance tests do not include proprietary ‘aspects’ Though this may well be done by a manufacturer with own

conformance tests for proprietary requirements


Conformance Testing Documentation and Process: ISO 9646 Methodology

Test Suite (Test Cases)

ATS

Req. checklist

ICS

Test Purposes

TSS & TP

testingTest

Report

logging and

analysis

Test System

Base Standard

(or Profile)

implementation

Product

Implementation Under Test

compilation

Executable Test Suite

(e.g., C++)

Industry

validation


Implementation Conformance Statement (ICS)

The ICS is a checklist of the capabilities supported by the Implementation Under Test (IUT)

Provides an overview of the features and options that are implemented in any given product

The ICS can be used to select and parameterise test cases and can be used as an indicator of basic interoperability between different products.

Conditional support e.g., Qn must be supported if Q1 supported then otherwise not.

Q# ICS Question Ref Status Support

Q1 Support of Feature F1 [x] Clause a OPTIONAL

Q2 Support of Feature F2 [x] Clause b MANDATORY

: : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL

: : : :

Qk Support of message Parameter P1 [y] Clause d OPTIONAL


Profile ICS

Q# ICS Question Ref Status Profile Status Support

Q1 Support of feature F1 [x] Clause a OPTIONAL MANDATORY

Q2 Support of feature F2 [x] Clause b MANDATORY MANDATORY

: : : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL MANDATORY

: : : : :

Qk Support of parameter P1 [y] Clause d OPTIONAL N/A

A profile ICS reflects how a (protocol) profile restricts the scope of a base standard by making certain options mandatory or excluding certain options.


Selecting Tests with the ICS

Q# ICS Question Ref Status Support

Q1 Support of Feature F1 [x] Clause a OPTIONAL

Q2 Support of Feature F2 [x] Clause b MANDATORY

: : : :

Qn Support of Message M1 [y] Clause c CONDITIONAL

: : : :

Qk Support of message Parameter P1 [y] Clause d OPTIONAL

A filled-in ICS can be used to select tests. For example, a test for an OPTIONAL feature which is not supported in a given product will be deselected from the test suite.


Parameterizing Tests with eXtra Information for Testing (IXIT)

Q# IXIT Question Ref Allowed Values Value

Q1 Network address [x] Clause a Valid IP address 12.34.56.76

Q2 Value of Timer T1 [x] Clause b 1 .. 7 s 5s

: : : : :

A filled-in IXIT can be used to parameterize tests. The Value column is used to specify explicit IUT or test system dependent values.


Test Purposes (TP)

Test Purposes (TP) are natural language, precise descriptions of the purpose of the test in relation to a particular (base standard) requirement

They are very focussed. They are not code.

A TP is WHAT to test not HOW to test

TPId: SIP_SS _PR_CE _V_012Selection: Mandatory [Q2]Precondition: Registration of both simulated UA to the IUT and an initiated sessionRef: 5.1.1 [1], 17.3.6 [1], 17.4 [1] , 10.46.6 [1]Purpose: Ensure that the IUT on receipt of a Server Internal Error (500 Server Internal Error) server failure response, sends an ACK request, with the same field Via and Branch parameter as in the previous INVITE, to the UAS and forwards it to the UAC.


Test Suite Structure (TSS)

RegistrationRegistrant

ValidInvalid

Registrar:

SessionOriginating Endpoint

Call EstablishmentTP_S_OE_CE_01TP_S_OE_CE_02TP_S_OE_CE_03:

Call Release :

Terminating EndpointProxyRedirect Server

:

Test Purposes are grouped into a logical Test Suite Structure (TSS) according to suitable criteria

e.g., basic interconnection, error handling, functionality etc.).

The complete document is usually called the Test Suite Structure and Test Purposes (TSS&TP)


Abstract Test Suite

A Test Case is the ‘implementation’ of the test purpose Is the HOW to test not WHAT to test

The Test Suite (ATS) is the collection of all Test Cases Most ETSI Test Suites are written in TTCN

Predominantly in TTCN-2 Progression to TTCN-3 for new work Not RF testing (generally not physical layer)

TTCN-3 Testing and Test Control Notation Allows tests to be specified in detail (test code) that is

independent of the (eventual) real test system on which it may be run

• i.e., TTCN-3 will run on any test system that supports a standardised TTCN-3 execution environment


Executable Test Suites

Executable test suites are compiled from the abstract Either direct to binary, or More commonly to some intermediate programming

language (C++, Java, etc.)

ETSI does not deliver executables, but We try to ensure all code can be compiled And validated by executing the tests against a real

implementation on at least one test system• 5 test systems in the case of UMTS, for example


Interoperability Testing End-to-End Interoperability of devices

What is being tested? Assignment of verdicts?

Need to distinguish between Formal interoperability testing And informal interoperability events used to validate/develop

technologies

Device1 Devicen

Device2


SUT

API

MMI

L3

L2

PHY

API

MMI

L3

L2

PHY

QE = Qualified

Equipment

EUT = Equipment Under Test

Interoperability Testing Tests Functionality Between Devices

Conformance Monitoring

Interoperability Testing(of terminals)

1 2 3

4 5 67 8 9

* 8 #

QE EUT

1 2 3

4 5 67 8 9

* 8 #

Means of Communication (MoC)


Characteristics of Interoperability Testing

Is system testing Tests a complete device or a collection of devices

Shows that (two) devices interoperate Albeit within a limited scenario

Tests at a ‘high’ level (as perceived by users) Tests the ‘whole’, not the parts

• e.g, protocol stacks + applications Tests functionality

• Shows function is accomplished (but not how)

Does not necessarily require a test system Uses existing interfaces (standard/proprietary)

Interoperability Testing is BROAD and SHALLOW Less thorough but wide in scope Gives a high-level of confidence that devices (or components in

a system) will interoperate with other devices (components)


Limitations of Interoperability Testing

Does not prove interoperability with other implementations with which no testing has been done A may interoperate with B and B may interoperate with C. But it

doesn’t necessarily follow that A will interoperate with C. Combinatorial explosion

Does not prove that a device is conformant Interoperable devices may still interoperate even though they

are non-conformant

Cannot explicitly test error behaviour or unusual scenarios Or other conditions that may need to be forced (lack of

controllability) Has limited coverage (does not fully exercise the device)

Not usually automated and may not be repeatable


Interoperability Test Methodology

ETSI TS 102 237-1

Uses best principles of ISO 9646 and adapts them for interoperability testing

Definitions

EUT – Equipemt Under Test

QE – Qualified Equipment

Concepts

IFS (like the PICS)

• Interoperable Functions Statement

Generic testing architecture

Test Descriptions (in prose, tabular format)

If suitable interfaces (API/MMI) are exposed then can automate the tests by running TTCN-3 scripts

Processes

Basic test specification development process


Interoperability Testing Documentation and Process: TS 102 237 Methodology

Interop Test Cases

Test Report

logging and

analysis

Base Standard (or Profile)

implementation

Product 1

Qualified Equipment

Product 2

Equipment Under Test

implementation

IFS (functional checklist)

Interop Test Purposes

testing


Test Purpose Language (TPLan)

ETSI language, not necessarily restricted to IOP testing Keywords and syntax provide clear and consistent

structure Keywords chosen for communications applications (sends,

receives etc.) Text between keywords not part of syntax so free

expression possible A TP’s basic structure (corresponding keyword):

Header (TP id) Pre-conditions (with) Stimulus (when) Expected response (then)


TP id : TP_COR_1719_02Summary : 'EUT sends packet to All-RT LL M/cast address'RQ ref : RQ_COR_1719Config : CF_021_ITD ref : TD_COR_1719_02

with { QE1 'configured as a router' and QE2 'configured as a router'}

ensure that { when { EUT is requested to 'send packet to All-Routers Link-Local M/cast addr' } then { QE1 indicates 'receipt of the packet' and QE2 indicates 'receipt of the packet' } }

TPLan Example for Interoperability


Interoperability Test Descriptions

Specify detailed steps to be followed to achieve stated test purpose

Steps are specified clearly and unambiguously without unreasonable restrictions on actual method: Example:

• Answer incoming call

NOT• Pick up telephone handset

Written in a structured and tabulated natural language so tests can be performed manually

Can be automated using TTCN‑3 when EUT has software interfaces


Example Test Description

Test: COR_001_02 Selection Criteria: Mandatory Selected: Yes

Title: Auto configure QE using a unique address in a simple network

Test Purpose: ensure that {when { etc……….

Pre‑test conditions: Configure the equipment to ensure that QE1 will not intend to use the same link-local address than the one used by EUT (EUT and QE1 interfaces carry different Interface identifiers and both run stateless autoconfig, or EUT Link-Local address configured manually).

EUT interface is up and carries a link-local address QE1 interface is down

Step Test description Verdict

Pass Fail

1 Turn on QE1 interface and wait a few seconds

2 EUT sends an echo request to the Link-Local address of QE1

3 Check: does EUT receive an echo reply from QE1? Yes No

4 QE1 sends an echo request to the Link-Local address of EUT

5 Check: does QE1 receive an echo reply from EUT? Yes No

Observations:


Automated IOP Testing Using TTCN-3

TestDriver

(TTCN-3)

TestDriver

(TTCN-3)

TestController(TTCN-3)

AT Command

s

AT Command

s


Network Integration Testing (NIT)

End-to-end testing of the system Components in the network must interoperate

Test Driver2

SUT

SIP

RADIUS

DIAMETER

PoC

Test Driver1


Example NIT for IMS

End-to-End Methodology

Mw Mw

UE1

Um Um

UE2

P-CSCF1

S-CSCF1 S-CSCF2

Cx

Mw

Access Network

HSS

I-CSCF P-CSCF2 Access Network


TTCN-3 can be used for Conformance and Interoperability Testing

A

TTCN-3

latigid

Conformance testing(of a network element)

1 2 3

4 5 6

7 8 9

* 8 #

Conformance testing(of terminal equipment)

latigid

TTCN-3

B

Interoperability testing(of terninal equipment)

1 2 3

4 5 6

7 8 9

* 8 #

1 2 3

4 5 6

7 8 9

* 8 #

TTCN-3TTCN-3


Benefits of TTCN-3

Specifically designed for testing

Concentrates on the test not the test system

Commonly understood syntax and operational semantics

Constantly maintained and developed

Off-the-shelf tools and TTCN-based test systems are readily available

Single language for many (all?) testing activities

Education and training costs can be rationalized

Maintenance of test suites (and products) is easier

Allows the application of a common methodology and style, both on a corporate level and within standardization


Main Capabilities of TTCN-3

Dynamic concurrent testing configurations

Various communication mechanisms (synch and asynch)

Data and signature templates with powerful matching mechanisms (including regular expressions)

Specification of encoding information

Display and user-defined attributes

Test suite parameterization

Control of Test Case execution and selection mechanisms

Control of complex test configurations

Assignment and handling of test verdicts

Harmonized with ASN.1 (XML and IDL coming)

Different presentation formats

Well-defined syntax, static - and operational semantics


The Core Languageand Other Presentation Formats

TTCN-3 Core Language

Text format

PresentationFormat3

PresentationFormatn

Graphical Format

Tabular Format

Core format is text based (most popular)

TTCN-3 can be edited or viewed in other formats

Tabular format (for TTCN-2 people)

Graphical format (good for visual overview)

Other standardized formats in the future?

Proprietary formats possible


Example Core (Text) Format

function PO49901(integer FL) runs on MyMTC {

L0.send(A_RL3(FL,CREF1,16))TAC.startalt {

[] L0.receive(A_RC1((FL+1) mod 2)) { TAC.cancel

verdict.set(pass)

}

[] TAC.timeout { verdict.set(inconc)

}

[] any.receive { verdict.set(fail) }

}}


Example Graphical Format


Example Tabular Format

Test Case Definition

Name : MyTestcase

Group :

Purpose : Example Testcase

System I/f :

MTC Type : MyComponentType

Comments :

Name Type Initial Value Comments

MyVar INTEGER 0

Behaviour Definition Comments alt { [ ] MyPort.receive(Msg); [ ] : }

DetailedComments:


Use of TTCN-3 With Other Languages

TTCN can be integrated with other languages

Fully harmonized with ASN.1 (1997)

Harmonized with other languages

IDL, XML, C/C++

TTCN-3 Core Language

IDL Types & Values

Other types & Valuesn

XML Types & Values

ASN.1 Types & Values


Test Suite

Major Elements of a TTCN-3 Test Suite

Test Behaviour

Test System Architecture

Actual Test Data

Test Data Types

Definition of data types for• Protocol Messages and• Information elements (fields, parameters)• Internal data (e.g., for computation)• Test coordination messages etc.Predefined simple types Integer, boolean, float bitstring, hexstring, octetstring charstring, universal charstring... and complex types record, record of, set, set of union, enumerated

... and special types such asverdict, address

Actual test data (values) used during testing

• Message templates• (Remote) signature templates

• Matching mechanisms

• values, lists, wildcards, • regular expressions etc.

• Encoding information

Test components

Test ports

Connections

• between test components

• to System Under Test

Configurations

• static

• dynamic

test cases

test steps

default behaviour

management of test components

sending/receiving messages

verdict assignment

computation (e.g., checksums)

timers and timeouts

test execution and control

etc.


Simple TTCN-3 Architecture

IUTTest Systemstimulus

response

Point of Control and Observation

(PCO)

Test port Test port

Implementation Under TestBlack-Box

TTCN-3 based Test

System


TTCN-3 Architecture – Test Components

Test System

Parallel Test Component

(PTC)

Parallel Test Component

(PTC)

Main Test Component

(MTC)

Communication to IUT

Internal Communication

IUT


TTCN-3 Module

module Example{

// Definitions part

control{

// Control part}

} with {encode "BER"}

Attributes

Module (…)

Module Control

Module Definitions


module Example{

group TestArchitecture{

// Port and Test Component definitions }group MessageTypes{

// Defintions of message types }group ActualMessages{

// Templates for instances of actual messages}group TestCases{

// Test Case definitions}

}

Definitions Partmodule Example{

// Port and Test Component definitions // Defintions of message types // Templates for instances of actual messages// Test Case definitions

}


Specifying Test Messages

type record MsgType{ integer msgId charstring msgName, charstring msgInfo}

template MsgType INVITE{ msgId 0 msgName "INVITE", msgInfo "For a good Brazilian Dinner"}

template MsgType ACCEPT{ msgId 1 msgName "ACCEPT", msgInfo *}


TTCN-3 Behaviour

INVITE ACCEPT or DECLINE

P

testcase TC1() runs on PTC1{

P.send(INVITE)T1.start(30E3)alt{

[] PCO.receive(ACCEPT){verdict.set(pass)}

[] PCO.receive(DECLINE){verdict.set(inconc)} [] PCO.receive{verdict.set(fail)}

[] T1.timeout{verdict.set(fail)}

}}


Attributes

The Control Part

Module (…)

Module Control

Module Definitions

module Example{

// Definitions part

params { boolean Q1;:

}

control{

if Q1 then execute (TC1()):

}} with {encode "BER"}


TTCN-3 Test System

ENCODER

DECODER

(N-protocol specific)

Adaptation Layers

TRITTCN-3 Runtime Interface

TCI TTCN-3 Control Interface

The Industry

Test Suite in TTCN-3

(source)

TTCN-3 Test Suite

(object)

Compilation

PICS etc

(reqs. catalogu

e)

Parameterisation

Selection

Underlying Protocol Stack

(N-1)

Connection to the SUT

Control / Logging

User


TTCN-3 Standardshttp://www.ttcn-3.org

ES 201 873-1 (Z.140) TTCN-3 Core Language

ES 201 873-2 (Z.141) TTCN-3 Tabular Presentation Format (TFT)

TR 101 873-3 (will eventually be ES 201 873-3) (Z.142) TTCN-3 Graphical Presentation Format (GFT)

ES 201 873-4 (Z.143) TTCN-3 Operational Semantics

ES 201 873-5 TTCN-3 Runtime Interface (TRI)

ES 201 873-6 TTCN-3 Control Interfaces (TCI)

ES 201 873-7 and upwards Using ASN.1, XML, IDL, C/C++, with TTCN-3


Concluding Thoughts (1)

Bad publicity in trade magazines Embarrassment for the manufacturer Annoyance of the end customer

In the past, interoperability failures meant:

Today, interoperability failures in the field means:

Front page headlines in the Financial Times Fall in manufacturers stock price Loss of investor confidence Unrecoverable damage to brand name Irretrievable loss of customers

We can no longer afford to get it wrong! Testing We can no longer afford to get it wrong! Testing is part of ETSI’s strategy to get it right!is part of ETSI’s strategy to get it right!


Concluding Thoughts (2)

ETSI membership believes in methodical testing A good example

GSM/UMTS is growing by 1 Million users per day! Can you imagine what would happen if users were

experiencing interoperability problems ? More than 1 Million Euros invested per year for writing

formal test specifications and test suites We have experienced what it means to get it right!

A bad example We have not always got it right! For GPRS we did not take a formal approach to testing First market products did not interoperate

We have learnt that testing is not cheap, but it pays in the long run

Thank You!

Anthony Wiles

ETSI Protocol and Testing Competence Centre

Phone +33 (0)4 92 94 43 26

e-mail [email protected]

Website www.etsi.org/ptcc

overview of etsi testing methodology ict-osa/parlay workshop brazil, march 2006

Documents