model-based testing principles

Università degli Studi dell’Aquila

L22. Model-based Testing (Principles)

Henry Muccini

DISIM, University of L’Aquila

www.henrymuccini.com, henry.muccini@univaq.it

Copyright Notice

The material in these slides may be freely reproduced

and distributed, partially or totally, as far as an explicit

reference or acknowledge to the material author is

preserved.

Henry Muccini

Agenda

Modeling

MBT:

→Why?

→Models for MBT

→UML-based Testing

─ Testing activities

Modeling

Model-based testing

Why?

Models for MBT

UML-based Testing

AGENDA

UML-based Testing

Modeling: there is not just programming

What is a model:

→A model is a simplification of the reality

→The reality is abstracted, since it is too complex

Why modelingWhy modeling

→Models are built for a better understanding of the

system under development

Everything is a model ?!?

Non-software Models …

Specification: What

Software Models

Other Software Models (Web models)

… other software models…

… and many others…

Pilot Multifunction

Display1

Pilot Multifunction

Display2

CoPilot Multifunction

Display1

CoPilot Multifunction

Display2

Display Processor

Display Processor

Display Processor

Display Processor

MissionProcessor

MissionProcessor

MissionProcessor

High speed network

1553 bus

Auto-Pilot GPS Nav Radio

1553 bus

High speed network

Model Driven Development (MDD)

Models are becoming first core assets in

Software Engineering/Architecting:

→models are primary artifacts retained as first class

entities that can be analyzed and manipulated by

means of automated toolsmeans of automated tools

→the focus moves from writing and sharing

documents to writing and sharing models.

From Document Driven to Model Driven

Models in MDD

Models in MDD can be used for:

→Documenting design decisions

→Guiding the coding phaseGuiding the coding phase

→For analyzing and validating design choices

─ by complementing traditional code-level

analysis techniques

Why modeling? For Documenting and Analysis

What is MBT?

It consists in:

→Extracting Test Cases from the Model

→Run the test case over the system implementation

Purpose:

→To validate the implementation conformance to the

model

─ The model itself is the oracle, i.e., it represents the

expected behavior

Model-based Testing

A model based testing approach

accepts two main inputs

→a model of the software under test,

→a set of test generation directives which guide the test cases

selection selection

and outputs a test specification

→which includes a set of stimuli the tester should introduce in

the system together with expected responses

Modeling for testing

Essentially, we need (at least) a behavioral model of

the software system:

→Labelled Transition System

→StateCharts

→UML State of Sequence Diagrams→UML State of Sequence Diagrams

→IOLTS

→Finite State Automata

→…

The base-line technique for designing test cases

→Timely

─ Often useful in refining specifications and assessing testability before code is written

→Effective

─ finds some classes of fault (e.g., missing logic) that can elude ─ finds some classes of fault (e.g., missing logic) that can elude other approaches

→Widely applicable

─ to any description of program behavior serving as spec

─ at any level of granularity from module to system testing.

→Economical

─ typically less expensive to design and execute than structural (code-based) test cases

(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

Program code is not necessary

→Only a description of intended behavior is needed

→Even incomplete and informal specifications can be used

─ Although precise, complete specifications lead to better test suites

Early functional test design has side benefits

→Often reveals ambiguities and inconsistency in spec

→Useful for assessing testability

─ And improving test schedule and budget by improving spec

→Useful explanation of specification

─ or in the extreme case (as in XP), test cases are the spec

(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

Different testing strategies (functional, structural,

fault-based, model-based) are most effective for

different classes of faults

Functional testing is best for missing logic faultsFunctional testing is best for missing logic faults

→A common problem: Some program logic was simply

forgotten

→Structural (code-based) testing will never focus on code that

isn’t there!

(C) 2007 MAURO PEZZÈ & MICHAL YOUNG

Modeling for testing

Such a model can be produced:

1. Through a formal specification language

─ Typically referred as formal testing

2. Through a diagrammatic (visual) notation

─ Typically referred as UML-based Testing

1. MBT based on Formal Specifications

Since the 80’s, many specification-based testing approaches have been proposed, based on formal languages such as

→Z, VDM, CSP, CCS, LOTOS, SDL, and Petri Nets.

More recently:

→Based on dynamic behavior:

─ FSM-based (Bochmann&Petrenko, Lee&Yannakis, …)

─ LTS-based (Brinksma&Tretmans, Jard&Jeron …)

→Focussing on static aspects,

─ ADT theory (Bernot&Gaudel…)

─ Z-based (Hierons…)

Formal Specification-based Testing

TorX (Côte de Resyste)

→ on-the-fly test generation and execution

→ random

→ LOTOS and Promela

TGV (IRISA - Rennes)TGV (IRISA - Rennes)

→ derives tests in TTCN from LOTOS or SDL

→ uses test purposes

TVEDA (CNET - France Telecom)

→ derives TTCN tests from SDL specification

2. MBT based on the UML

UML State-machine based testing:

→which might require the translation of the UML

diagrams into an intermediate formal description

Scenario-based testing:

→based on the use of opportunely annotated UML Sequence Diagram→based on the use of opportunely annotated UML Sequence Diagram

State-based and Scenario-based:

combine the use of interaction diagrams and state diagram,

possibly annotated with further (formal) information (e.g. OCL)

Model-based Testing: the overall idea

From Leila Naslavsky Advancement report

Model-based Testing Approaches

TESTO

UMLAU

TOTEM

SOOT

COW-SU

SCENTO

SeDiTe

AGEDI

UMLTe

UCSC-Sy

AsmLOR

UT

MTF

UITE

OR

eC IS

est

ystem

L

Main Testing Activities

Test Selection/Generation:

→it consists in selecting a suitable and finite set of test cases

from the possibly infinite set;

Test Execution and Evaluation: Test Execution and Evaluation:

→it consists in executing the code accordingly to the selected

test cases and comparing real and expected results;

Test Selection

A test case, in MBT, is typically a possible

“scenario”

→In scenario-based testing:

─ A scenario itself is an abstract test case

→In state-based testing

─ Execution flows are extraced from the state machine

and represent a test case

─ MB coverage criteria are applied on this model

Test Case Selection

Abstract Test case = path over the LTS/FSM,

corresponding to sequences of events

─ They are not “concrete” test cases, but abstract

(depending on the specification)

→Concrete Test cases = mapping asbtract test cases

into execution statements to be launched

Test Case Selection

inSD

C1 C2

m9

m1

Model of the systemTest

directives

11

outSD1

C1 C2 C3

m9m3

m8m1

Test specification

2

Test Case Selection

Test Execution

It consists in running the test cases on the system

implementation

→So to compare the real execution with the expected behavior

MBT practical concerns

Traceability, i.e. relating the abstract values of the specification to the concrete values of the implementation.

Execution, i.e. forcing the Implementation Under Test Execution, i.e. forcing the Implementation Under Test (IUT) to execute the specific sequence of events thathas been selected.

Testing and Industry

In order to be suitable for industrial needs a testingapproach has to emphasize the following qualities:

→Usability

─ Not ad-hoc models

→Timeliness

─ Soon and even incompleteOpposed to

─ Soon and even incomplete

→Tool support

Goal of a suitable testing approach should be“to improve the test results accuracy without sensiblyraising the testing effort”

Opposed to accuracy

MBT Tools

There are some automated tools for MBT:

→Telelogic Tau and Tau Tester

→Leirios Test Designer

→Conformiq Qtronic

→Rhapsody Test Conductor and Automatic Test Generation→Rhapsody Test Conductor and Automatic Test Generation

→AGEDIS

MBT in industry: challenges and issues

As remarked by Alan Hartman in his 2006 speech at ISSTA:

→“Only 1% of industry uses it”

Why?

Why?

The main reasons I foresee:

→Model2Code (and viceversa) traceability problems

→modeling is not yet so widely recognized as part of the development process.

─ Legacy systems─ Legacy systems

→because of tool/service problems

→cost/effectiveness is still under exploration

Conformiq: Two models are required in Input

•SutModel.Java (in QML)

•SutModel.xmi (Graphical Notation)

QML

Just like Java, QML consists of variables and

expressions, all strictly typed and known before the

compilation.

The types are divided into the same three groups that The types are divided into the same three groups that

Java has: primitives, references and values

QML

QML extends Java with:

Records

Ports

System blockSystem block

State machines

SutModel Graphical

SutModel Graphical

Coverage

Figure 5 – The Coverage Editor window