overview of etsi testing methodology...overview of etsi testing methodology anthony wiles manager...

Overview of ETSI Testing Methodology

Anthony Wiles

Manager

ETSI Protocol and Testing Competence Centre

ETSI Testing Methodology - SINTESIO Seminar 2006 2

ETSI

European Telecommunications

Standards Institute

Sophia Antipolis, France

The home of ICT standardization

http://www.etsi.org


Interoperability is …

� The ultimate aim of ICT standardisation

� The red thread running through the entire standards development process, it’s not an isolated issue

� Not something to be somehow fixed at the end

� The ability of systems and products to interwork is fundamental

� ETSI approach� ETSI produces with the required degree of parallelism

• Base Standards and

• Test Specifications

� Base standards should be designed with interoperability in mind

� Profiles designed to reduce potential non-interoperability

� Protocol conformance and Interoperability statements

� Two complementary forms of testing• Conformance testing

• Interoperability testing (formal IOT)

� Testing provides vital feedback into the standardization work


Poor Interoperability is Expensive

� Bad publicity in trade magazines

� Embarrassment for the manufacturer

� Annoyance of the end customer

In the past, interoperability failures meant:

Today, interoperability failures in the field means:

� Front page headlines in the Financial Times

� Fall in manufacturers stock price

� Loss of investor confidence

� Unrecoverable damage to brand name

� Irretrievable loss of customers

We can no longer afford to get it wrong! We can no longer afford to get it wrong!


Specification and Testing is Part of

ETSI’s Strategy (to get it right!)

� ETSI membership believes in methodical testing

� A good example

� GSM/UMTS is growing by 1 Million users per day!

� Can you imagine what would happen if users were experiencing interoperability problems ?

� More than 1 Million Euros invested per year for writing formal test specifications

� We have experienced what it means to get it right!

� A bad example

� We have not always got it right!

� For GPRS we did not take a formal approach to testing

� First market products did not interoperate

� We have learnt that testing is not cheap, but it pays in the long run


� Technical Committee MTS (Methods for Testing and Specification)

�Development of methodologies, techniques and languages

(including TTCN3)

• http://portal.etsi.org

� ETSI’s Protocol and Testing Competence Centre (PTCC)

�Supports ETSI committees on the application of formal

techniques in standards on a daily basis

�Development of test specifications (conformance and interop)

• http://portal.etsi.org/ptcc/

� ETSI Plugtests™ Service

�Validation of standards and prototypes through

interoperability events

• http://www.etsi.org/plugtests

ETSI Resources for Interoperability


How Does the PTCC Help?

� Assist ETSI Technical Bodies on the use of state of

the art techniques for

� Specification, validation and testing

� Good working practices (Standards Engineering)

� Pragmatic and flexible approach

� Based on experience

� Help to develop usable methodologies

� For ETSI’s current and emerging needs

� Knowledge transfer

� Quality through Continuity


Who PTCC Supports

� Technical Bodies (TB)

� Technical Committees

� ETSI Projects

� Partnership Projects etc.

� Chairmen, Rapporteurs, Individuals

� Working Groups (WG)

� STFs (Specialist Task Forces)

� Experts are seconded from the ETSI membership

• 25-30 experts in 2006

� Produce test specifications (including validation)

� Short projects

• e.g., 2 man-months maintenance of VoIP tests

� Long projects

• e.g., UMTS testing 58 man-months per year over 3-5 years

� 15 – 20 PTCC STFs per year

� Responsible for approx. 2,7M€ of expert resource per year


Typical Standards Development

Process in ETSI

(Unit) Conformance Testing

Interoperability Testing

Products mature from prototypes to commercial products

ETSI: Development of base standards

Certification

IndustryIndustryIndustryIndustry

time

Conformance Test Specifications

Interoperability Test SpecificationsIterative

feedback

Iterative

feedback


PTCC Support for Specification Techniques

� PTCC provides support to the ETSI TBs on the use of

modern specification techniques in ETSI deliverables:

� UML (Unified Modeling Language)

� ASN.1 (Abstract Syntax Notation)

� XML (Extensible Markup Language)

� MSC (Message Sequence Charts)

� SDL (Specification and Description Language)

� Etc.

� TTCN (Test and Test Control Notation)


Different Kinds of ETSI Test

Specifications

Conformance Robustness

PerformanceInteroperability

Network

IntegrationRF/EMC


Typical ETSI PTCC Areas of Testing

� Cellular: GSM and 3G (UMTS) terminals

� Now includes early IMS

� WiFi: HiperMAN, HiperACCESS, WiMax

� VoIP: H.323, SIP, SIGTRAN

� Service Creation: OSA/Parlay (API, IDL)

� IPv6: Core, Security, Mobility, v4-v6

� Cordless phones: DECT

� Radio communications: TETRA, DMR

� Access terminals: FSK, SMS

� Broadband: ISDN, DSL

� Smartcards: Readers, cards, security modules

� Intelligent Transport Systems (ITS): DSRC

� Early NGN

� Future: More Security, more NGN, GRID ...


Progressive Testing

� All engineering disciplines accept that testing should be applied to progressively complex units� From individual components to entire systems

� This demands a number of different testing solutions� There is no silver bullet

� In the two extremes � Just testing the components is not enough

� Just testing the system is not enough

� Limitations are economic, not technical� What can I afford to test?

� What can I afford not to test?

� What should be covered by standardised test specifications

� What is the right kind of testing for the job� What kind of ‘thing’ is being tested? Device? System? Service?

� What is the most economic method(s)?

� What resources are (or must be made) available?• Tools, test scripts, testbeds etc.


Conformance Testing

� Is Black-Box testing

� Stimulation and Response

Tester Device

Point of Control and Observation (PCO)

Reqs.


Conformance Testing Tests is Usually

Layer-byLayer

lat

ii

1 2 3

4 5 6

7 8 9

* 8 #

latigid

Test

SystemSUT

API

MMI

L3

L2

PHY

SUT = System Under Test

IUT = Implementation Under Test

L3

Upper TesterUpper TesterUpper TesterUpper Tester

Lower TesterLower TesterLower TesterLower Tester


Characteristics of Conformance

Testing (1)

� Is unit testing

� Tests a single ‘part’ of a device (e.g., a protocol layer)

• Often incrementally – layer-by-layer

� Tests against well-specified requirements

� For conformance to the requirements in a base specification or profile (standard)

� Usually limited to one requirement per test

� Tests at a 'low' level

� At the protocol (message/behaviour) level (bits)

• Or service primitive, API, Interface

� Tests over standardised interfaces

� May not be available to ‘normal’ user

� Requires a test system (and executable test cases)

� Can be expensive (e.g., radio-based test system)

� Tests under ideal conditions


Characteristics of Conformance Testing (2)

� High control and observability

� Means we can explicitly test error behaviour

� Can provoke and test non-normal (but legitimate)

scenarios

� Can be extended to include robustness tests

� It is usually automated and tests are repeatable

� Conformance Testing is DEEP and NARROW

� Thorough and accurate but limited in scope

� Gives a high-level of confidence that key components of

a device or system are working as they were specified

and designed to do


Limitations of Conformance Testing

� Does not prove end-to-end functionality

(interoperability) between communicating systems

� Conformance tested implementations may still not

interoperate

• This is often a specification problem rather than a testing

problem! Need minimum requirements or profiles

� Does not test a complete system

� Tests individual system components, not the whole

• A system is often greater than the sum of its parts!

• Does not test functionality

– Does not test the user’s ‘perception’ of the system

� Standardised conformance tests do not include

proprietary ‘aspects’

� Though this may well be done by a manufacturer with

own conformance tests for proprietary requirements


Conformance Testing Methodology

ISO/IEC 9646 (parts 1-7)

Test Suite

(Test Cases)

ATS

Req.

checklist

ICS

Test

Purposes

TSS & TP

testingTest

Report

logging and

analysisTest System

Base

Standard

(or Profile)

implementation

Product

Implementation

Under Test

compilation

Executable

Test Suite

(e.g., C++)

IndustryIndustryIndustryIndustry

validation


Implementation Conformance

Statement (ICS)

The ICS is a checklist of the capabilities supported by the Implementation

Under Test (IUT)

Provides an overview of the features and options that are implemented in any

given product

The ICS can be used to select and parameterise test cases and can be used

as an indicator of basic interoperability between different products.

Conditional support e.g., Qn must be supported if Q1 supported then

otherwise not.

CONDITIONAL[y] Clause cSupport of Message M1Qn

::::

::::

OPTIONAL[y] Clause dSupport of message Parameter P1Qk

MANDATORY[x] Clause bSupport of Feature F2Q2

OPTIONAL[x] Clause aSupport of Feature F1Q1

SupportStatusRefICS QuestionQ#


Profile ICS

N/A

:

MANDATORY

:

MANDATORY

MANDATORY

Profile Status

OPTIONAL[y] Clause dSupport of parameter P1Qk

::::

CONDITIONAL[y] Clause cSupport of Message M1Qn

::::

MANDATORY[x] Clause bSupport of feature F2Q2

OPTIONAL[x] Clause aSupport of feature F1Q1


A profile ICS reflects how a (protocol) profile restricts the

scope of a base standard by making certain options

mandatory or excluding certain options.


Selecting Tests with the ICS

��CONDITIONAL[y] Clause cSupport of Message M1Qn

::::

::::

��OPTIONAL[y] Clause dSupport of message Parameter P1Qk

��MANDATORY[x] Clause bSupport of Feature F2Q2

��OPTIONAL[x] Clause aSupport of Feature F1Q1


A filled-in ICS can be used to select tests. For example, a

test for an OPTIONAL feature which is not supported in a

given product will be deselected from the test suite.


Parameterizing Tests with eXtra

Information for Testing (IXIT)

:

5s

12.34.56.76

Value

::::

1 .. 7 s[x] Clause bValue of Timer T1Q2

Valid IP address[x] Clause aNetwork addressQ1

Allowed ValuesRefIXIT QuestionQ#

A filled-in IXIT can be used to parameterize tests. The

Value column is used to specify explicit IUT or test system

dependent values.


The Requirements Catalogue

� Each Requirement is categorized as follows (example from

IPv6):

� Requirement type:

• Mandatory (MUST, MUST NOT)

• Recommended (SHOULD, SHOULD NOT)

• Optional (MAY, MAY NOT)

� Requirement target:

• E.g., Host, Router, Node (Host or Router)

� Requirement text

� Functional groupings

• E.g., Process Fragmented packet, Generate ICMPv6 Error Typetc.

� A scalable database containing all requirement elements

� Web interface

� Browsing by function

� User-defined search

� Links to RFC and related test specification


ETSI IP Testing Library


Conformance Test Purposes

� Test Purposes (TP) are natural language, precise

descriptions of the purpose of the test in relation to a

particular (base standard) requirement

� Define the functionality being tested—the WHAT

� Do not define HOW to test the function

� Grouped into a logical structure (Test Suite Structure)

� TSS & TP

� One Requirement may spawn several TPs

� A conformance TP is written on the protocol level

� Specified in

� Natural language, or

� ETSI’s Test Purpose Language (TPLan)

� They are not test code


TPLan Example for Conformance

TP id : TP_COR_0047_01Summary : ‘hop limit of one'RQ Ref : RQ_COR_0047Config : CF_02_CTC Ref : TC_COR_0047_01ensure that {

--Stimuluswhen { IUT receives ‘Ipv6 packet' from ‘Host'

containing ‘IPv6 Header'indicating ‘Hop limit' set to ‘1‘ }

--Expected responsethen { IUT sends ‘ICMPv6 Time Exceeded' to ‘Host‘

containing ‘ICMP code' set to ‘ZERO‘ }}


Conformance Test Cases

� Detailed TTCN-3 test script that implements test purpose

� can be compiled and executed

� Is the HOW to test not WHAT to test

� Composition

� a preamble

� test body (i.e., implementation of the Test Purpose)

� A postamble

� Assigns test verdicts

� Handles unexpected behaviour as well as the behaviour in the test purpose

� Can be distributed over parallel test components

� Can be entirely automated

� Configurable at run-time, e.g., SUT address


Abstract Test Suite

� The Test Suite (ATS) is the collection of all Test Cases

� Most ETSI Test Suites are written in TTCN

� Predominantly in TTCN-2

� Progression to TTCN-3 for new work

� Not RF testing (generally not physical layer)

� TTCN-3

� Testing and Test Control Notation

� Allows tests to be specified in detail (test code) that is

independent of the (eventual) real test system on which

it may be run

• i.e., TTCN-3 will run on any test system that supports a

standardised TTCN-3 execution environment


What is TTCN-3?

� Internationally standardized testing language

� Look and feel of a regular programming language

� Allows unambiguous implementation of tests

� Independent of execution environment due to

separate test system interface standards

� Wide range of applications from mobile (radio)

communications to Internet to software

� Good tool support

� Good book:

� http://eu.wiley.com/WileyCDA/WileyTitle/productCd-

0470012242.html


Example TTCN-3 Test Case

testcase TC_COR_0047_01() runs on Ipv6Node system EtherNetAdapter {f_cf02Up(); // Configure test system for HS->RT

// No preamble required in this casef_TP_HopsSetToOne(); // Perform test

// No postamble required in this casef_cf02Down(); // Return test system to initial state

}

function f_TP_HopsSetToOne() runs on Ipv6Node {var Ipv6Packet v_ipPkt;var FncRetCode v_ret := f_echoTimeExceeded( 1, v_ipPkt );if ( v_ret == e_success and v_ipPkt.icmpCode == 0 ) { setverdict(pass);}else { setverdict(fail); }

}

function f_echoTimeExceeded(in UInt8 p_hops, out Ipv6Packet p_ípPkt ) runs on Ipv6Node return FncRetCode {

var Ipv6Packet v_ipPacket; var FncRetCode v_ret;ipPort.send( m_echoReqWithHops(p_hops) );alt {[] ipPort.receive( mw_anyTimeExceeded ) -> value p_ipPkt

{ return e_success }[] ipPort.receive { return e_error } }

}


Executable Test Suites

� Executable test suites are compiled from the abstract

� Either direct to binary, or

� More commonly to some intermediate programming

language (C++, Java, etc.)

� ETSI does not deliver executables, but

� We try to ensure all code can be compiled

� And validated by executing the tests against a real

implementation on at least one test system

• 5 test systems in the case of UMTS, for example


TTCN-3 Test System

ENCODERENCODERENCODERENCODER

DECODERDECODERDECODERDECODER

(N(N(N(N----protocol protocol protocol protocol specific)specific)specific)specific)

Adaptation LayersAdaptation LayersAdaptation LayersAdaptation LayersTRITRITRITRITTCN-3 Runtime Interface

TCI TCI TCI TCI TTCN-3 Control Interface

The IndustryThe IndustryThe IndustryThe Industry

Test Suite Test Suite Test Suite Test Suite in TTCNin TTCNin TTCNin TTCN----3333

(source)(source)(source)(source)

TTCNTTCNTTCNTTCN----3 3 3 3 Test SuiteTest SuiteTest SuiteTest Suite

(object)(object)(object)(object)Compilation

PICS etcPICS etcPICS etcPICS etc

((((reqsreqsreqsreqs. . . . catalogue)catalogue)catalogue)catalogue)

Parameterisation

Selection

Underlying Protocol StackUnderlying Protocol StackUnderlying Protocol StackUnderlying Protocol Stack

(N(N(N(N----1)1)1)1) Connection to the SUT

Control / LoggingControl / LoggingControl / LoggingControl / Logging

User


Interoperability Testing

� End-to-End Interoperability of devices

� What is being tested?

� Assignment of verdicts?

� Need to distinguish between

� Formal interoperability testing

� And informal interoperability events used to validate/develop technologies

Device1 Devicen

Device2


SUT

API

MMI

L3

L2

PHY

API

MMI

L3

L2

PHY

QE =

Qualified

Equipment

EUT =

Equipment

Under Test

Interoperability Testing Tests

Functionality Between Devices

Interoperability Testing(of terminals)

1 2 3

4 5 6

7 8 9

* 8 #

QE EUT

1 2 3

4 5 6

7 8 9

* 8 #

Means of Communication (MoC)


Characteristics of Interoperability

Testing

� Is system testing

� Tests a complete device or a collection of devices

� Shows that (two) devices interoperate

� Albeit within a limited scenario

� Tests at a ‘high’ level (as perceived by users)

� Tests the ‘whole’, not the parts

• e.g, protocol stacks + applications

� Tests functionality

• Shows function is accomplished (but not how)

� Does not necessarily require a test system

� Uses existing interfaces (standard/proprietary)

� Interoperability Testing is BROAD and SHALLOW

� Less thorough but wide in scope

� Gives a high-level of confidence that devices (or components

in a system) will interoperate with other devices (components)


Limitations of Interoperability Testing

� Does not prove interoperability with other implementations

with which no testing has been done

� A may interoperate with B and B may interoperate with C. But it

doesn’t necessarily follow that A will interoperate with C.

� Combinatorial explosion

� Does not prove that a device is conformant

� Interoperable devices may still interoperate even though they

are non-conformant

� Cannot explicitly test error behaviour or unusual scenarios

� Or other conditions that may need to be forced (lack of

controllability)

� Has limited coverage (does not fully exercise the device)

� Not usually automated and may not be repeatable


Interoperability Testing Methodology

ETSI TS 102 237

Interop Test

Cases

Test Report

logging and

analysis

Base Standard

(or Profile)

implementation

Product 1

Qualified

Equipment

Product 2

Equipment

Under Test

implementationIFS (functional

checklist)

Interop Test

Purposes

testing


Interoperability Test Purposes

� Define the function being tested—the WHAT

� Do not define HOW to test the function

� Grouped into a logical structure (Test Suite Structure)

� One Requirement may spawn several TPs

� An interoperability TP is on the functional level

� Specified in ETSI’s Test Purpose Language (TPLan)


TP id : TP_COR_1719_02Summary : 'EUT sends packet to All-RT LL M/cast address'RQ ref : RQ_COR_1719Config : CF_021_ITD ref : TD_COR_1719_02

with { QE1 'configured as a router'

and QE2 'configured as a router'}

ensure that {when { EUT is requested to

'send packet to All-Routers Link-Local M/cast addr' } then { QE1 indicates 'receipt of the packet'

and QE2 indicates 'receipt of the packet' }}

TPLan Example for Interoperability


Interoperability Test Descriptions

� Specify detailed steps to be followed to achieve

stated test purpose

� Steps are specified clearly and unambiguously

without unreasonable restrictions on actual method:

� Example:

• Answer incoming call

NOT

• Pick up telephone handset

� Written in a structured and tabulated natural

language so tests can be performed manually


Example Interoperability Test Description

Identifier:

Step Step

Test Purpose: TP_COR_1100_01 Reference: RQ_COR_1100 Configuration:

TD_COR_1100_01

Summary EUT reassembles a fragmented packet of an original length less than 1500 octets

with { 'the MTU on Link1 set to 1400 octets' }

ensure that {when { QE is requested to 'send data requiring a packet

length greater than 1500 octets' }

then { EUT indicates 'receipt of the same data without modification' }

}

Pre-Test

Conditions:MTU set to 1400 octets on link1

Verdict

1

2

3

Cause QE to send an Echo Request to EUT with a

packet size of 1450 octets and with each octet set

to the hexadecimal value "F0"

CF_011_I

Check: Does protocol monitor show that the Echo

Request was sent from QE to EUT?Yes No

Check: Does QE receive an Echo Reply from EUT

with the packet length the same as the

Echo Request and with each octet

containing the hexadecimal value "F0"?

Yes No

Pass Fail

Test Description

Observations


Automated IOP Testing Using TTCN-3

TestDriver

(TTCN-3)

TestDriver

(TTCN-3)

TestController(TTCN-3)

AT Commands

AT Commands


IOT with Conformance Verifiction

(aka NIT - Network Integration Testing)


Example NIT for IMS

End-to-End Testing

Mw Mw

UE1

Um Um

UE2

P-CSCF1

S-CSCF1 S-CSCF2

Cx

Mw

Access Network

HSS

I-CSCF P-CSCF2

Access Network


Conformance and Interoperability

Testing are Complementary

� ETSI experience

� As you move up a system stack the emphasis should change from conformance to IOT

� Lower layer protocols, infrastructure

� Emphasis on conformance

� Middleware, enablers

� Combination of Conformance + IOT

� Services, applications, systems

� Emphasis on IOT

� Conformance testing as a pre-requisite to IOT

Interop TestingInterop TestingInterop TestingInterop TestingConformance Conformance Conformance Conformance TestingTestingTestingTesting

InteroperabilityInteroperabilityInteroperabilityInteroperability


TTCN-3 Standardshttp://www.ttcn-3.org

� ES 201 873-1 (Z.140)

� TTCN-3 Core Language

� ES 201 873-2 (Z.141)

� TTCN-3 Tabular Presentation Format (TFT)

� TR 101 873-3 (will eventually be ES 201 873-3) (Z.142)

� TTCN-3 Graphical Presentation Format (GFT)

� ES 201 873-4 (Z.143)

� TTCN-3 Operational Semantics

� ES 201 873-5

� TTCN-3 Runtime Interface (TRI)

� ES 201 873-6

� TTCN-3 Control Interfaces (TCI)

� ES 201 873-7 and upwards

� Using ASN.1, XML, IDL, C/C++, with TTCN-3


TTCN-3 can be used for Conformance

and Interoperability Testing

latigid

1 2 3

4 5 6

7 8 9

* 8 #

latigid

1 2 3

4 5 6

7 8 9

* 8 #

1 2 3

4 5 6

7 8 9

* 8 #


Benefits of TTCN-3

� Specifically designed for testing

� Concentrates on the test not the test system

� Commonly understood syntax and operational semantics

� Constantly maintained and developed

� Off-the-shelf tools and TTCN-based test systems are

readily available

� Single language for many (all?) testing activities

� Education and training costs can be rationalized

� Maintenance of test suites (and products) is easier

� Allows the application of a common methodology and

style, both on a corporate level and within

standardization


Main Capabilities of TTCN-3

� Dynamic concurrent testing configurations

� Various communication mechanisms (synch and asynch)

� Data and signature templates with powerful matching mechanisms (including regular expressions)

� Specification of encoding information

� Display and user-defined attributes

� Test suite parameterization

� Control of Test Case execution and selection mechanisms

� Control of complex test configurations

� Assignment and handling of test verdicts

� Harmonized with ASN.1 (XML and IDL coming)

� Different presentation formats

� Well-defined syntax, static - and operational semantics


The Core Language

and Other Presentation Formats

TTCN-3

Core

Language

Text format

Presentation

Format3

Presentation

Formatn

Graphical

Format

Tabular

Format

� Core format is text based

(most popular)

� TTCN-3 can be edited or

viewed in other formats

� Tabular format (for

TTCN-2 people)

� Graphical format (good

for visual overview)

� Other standardized

formats in the future?

� Proprietary formats

possible


Example Core (Text) Format

function PO49901(integer FL) runs on MyMTC{

L0.send(A_RL3(FL,CREF1,16))TAC.startalt {

[] L0.receive(A_RC1((FL+1) mod 2)) {TAC.cancel

verdict.set(pass)

}

[] TAC.timeout {verdict.set(inconc)

}

[] any.receive {verdict.set(fail)

}}

}


Example Graphical Format


Example Tabular Format

Test Case Definition

Name : MyTestcase

Group :

Purpose : Example Testcase

System I/f :

MTC Type : MyComponentType

Comments :

Name Type Initial Value Comments

MyVar INTEGER 0

Behaviour Definition Comments

alt

{ [ ] MyPort.receive(Msg);

[ ]

:

}

DetailedComments:


Use of TTCN-3 With Other Languages

� TTCN can be integrated with other languages

� Fully harmonized with ASN.1 (1997)

� Harmonized with other languages

� IDL, XML, C/C++

TTCN-3

Core

Language

IDL Types &

Values

Other types

& Valuesn

XML Types &

Values

ASN.1 Types

& Values


Test Suite

Major Elements of a TTCN-3 Test Suite

Test

Behaviour

Test System

Architecture

Actual

Test Data

Test Data

Types

Definition of data types for

• Protocol Protocol Protocol Protocol Messages andMessages andMessages andMessages and• Information elements (fields, parameters)Information elements (fields, parameters)Information elements (fields, parameters)Information elements (fields, parameters)• Internal data (e.g., for computation)Internal data (e.g., for computation)Internal data (e.g., for computation)Internal data (e.g., for computation)• Test coordination messages etc.Test coordination messages etc.Test coordination messages etc.Test coordination messages etc.Predefined simple typesIntegerIntegerIntegerInteger, , , , booleanbooleanbooleanboolean, , , , floatfloatfloatfloatbitstringbitstringbitstringbitstring, , , , hexstringhexstringhexstringhexstring, , , , octetstringoctetstringoctetstringoctetstringcharstringcharstringcharstringcharstring, , , , universaluniversaluniversaluniversal charstringcharstringcharstringcharstring... and complex typesrecordrecordrecordrecord, , , , recordrecordrecordrecord ofofofof, , , , setsetsetset,,,, set ofset ofset ofset ofunionunionunionunion, , , , enumeratedenumeratedenumeratedenumerated... and special types such as

verdict, addressverdict, addressverdict, addressverdict, address

Actual test data (values) used during

testing

• Message templates

• (Remote) signature templates

• Matching mechanisms

• values, lists, wildcards,

• regular expressions etc.

• Encoding information

Test components

Test ports

Connections

• between test components

• to System Under Test

Configurations

• static

• dynamic

test cases

test steps

default behaviour

management of test components

sending/receiving messages

verdict assignment

computation (e.g., checksums)

timers and timeouts

test execution and control

etc.


Simple TTCN-3 Architecture

IUTIUTIUTIUTTest SystemTest SystemTest SystemTest System

stimulus

response

PointPointPointPoint of ControlControlControlControl and

ObservationObservationObservationObservation (PCO)

Test port Test port

Implementation

Under Test

Black-Box

TTCN-3 based Test

System


TTCN-3 Architecture – Test Components

Test SystemTest SystemTest SystemTest System

Parallel Test Component

(PTC)

Parallel Test Component

(PTC)

Main Test Component

(MTC)

Communication to IUT

Internal Communication

IUTIUTIUTIUT


TTCN-3 Module

module Example{

// Definitions part

control{

// Control part}

} with {encode "BER"}

Attributes

Module (…)

Module

Control

Module

Definitions


module Example{

group TestArchitecture{

// Port and Test Component definitions}group MessageTypes{

// Defintions of message types }group ActualMessages{

// Templates for instances of actual messages}group TestCases{

// Test Case definitions}

}

Definitions Partmodule Example{

// Port and Test Component definitions// Defintions of message types // Templates for instances of actual messages// Test Case definitions

}


Specifying Test Messages

type record MsgType{ integer msgIdcharstring msgName,charstring msgInfo

}

template MsgType INVITE{ msgId 0msgName "INVITE",msgInfo "For a good Brazilian Dinner"

}

template MsgType ACCEPT{ msgId 1msgName "ACCEPT",msgInfo *

}


TTCN-3 Behaviour

INVITE ACCEPT

or

DECLINE

P

testcase TC1() runs on PTC1{

P.send(INVITE)T1.start(30E3)alt{

[] PCO.receive(ACCEPT){verdict.set(pass)}

[] PCO.receive(DECLINE){verdict.set(inconc)} [] PCO.receive{verdict.set(fail)}

[] T1.timeout{verdict.set(fail)}

}}


Attributes

The Control Part

Module (…)

Module

Control

Module

Definitions

module Example{

// Definitions part

params { boolean Q1;:

}

control{

if Q1 then execute (TC1()):

}} with {encode "BER"}

Thank You!

Phone +33 (0)4 92 94 43 26

e-mail [email protected]

Website www.etsi.org/ptcc

overview of etsi testing methodology...overview of etsi testing methodology anthony wiles manager...

Documents