asdspec, model based testing
Post on 17-Jan-2017
246 Views
Preview:
TRANSCRIPT
Rachid Kherrazi 6-4-2014 Grenoble
USING FORMAL SPECIFICATIONS TO SUPPORT MODEL BASED TESTING ASDSPEC: A TOOL COMBINING THE BEST OF TWO TECHNIQUES
PEOPLE INVOLVED
• Rachid Kherrazi
– Senior Consultant @ Nspyre
– Domain
• Process and Product Improvement
• RAMS (Reliability Availability Maintenance and Safety)
• Model Based Testing and Model Driven Engineering
• Also : Arjan van Der Meer & Marc Hamilton
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 2
ABOUT NSPYRE
MARKET SEGMENTS
• High Tech
• Traffic & Infra
• Industry
AREA’S OF EXCELLENCE
(Model Based) Systems Engineering /Model Driven
Engineering / Model Based Testing /Industrial Automation /
Simulation / Big Data / Mobile Solutions
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 3
CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 4
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
Model Driven System
Development
System Acceptance
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 5
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
Model Based System
Engineering
System Acceptance
Some of related technology
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 6
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
Model Driven
Engineering
System Acceptance
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Some of related technology
7
V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/
Requir
em
ents
Reposi
tory
System Acceptance
Model Based Testing
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Some of related technology
8
Integration Design
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
PAGE 9
V-MODEL FOR TESTING
3 main steps in test process
Module
Subsystem
System
9
MBT IS THE AUTOMATION OF TEST CASE GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Model
base
d t
est
ing
Tra
dit
ional
manual
test
ing
Tra
dit
ional
auto
mati
c t
est
ing
Manual
Scripting
Testing skills + Scripting skills + Modeling skills
Manual
Modeling
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 10
PERCEIVED BENEFITS
• Increased productivity (increased automation)
• Better test script maintenance
• Improved product reliability (new type of bugs, Increased test coverage)
• Agility ( Easily react to new feature changes, Reusability of test semantics, Early test engagement, Drive quality upstream)
• Increased employee satisfaction (challenging, new horizon, fun)
• Question: How can we improve MBT e.g. increase further the productivity?
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 11
MBT IS THE AUTOMATION OF TEST CASE GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Model
base
d t
est
ing
Tra
dit
ional
manual
test
ing
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 12
Testing skills
Manual
Modeling
MBT n
ext
Semi-automatic
Modeling
Manual
Scripting
+ Scripting skills Modeling skills
Tra
dit
ional
auto
mati
c t
est
ing
IDEA: USING BEST OF BOTH WORLDS
• Automatic generation of (partial) Spec Explorer test model from existing ASD interface model
Benefits
• Less effort for model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
• Results: High Quality, Reduced cost
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Statistical Dynamic
ASDspec
13
USE CASE: CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
• Multiple components
• Components need to interact to function
• Controller needed to coordinate interaction
14
Method
OVERVIEW OF PROJECT STAGES
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 15
Use case
/SUT
Goal
Technique
Tool
Results
Stage 1
Container Terminal (CT)
Development of control
software for the CT
Model Driven Engineering +
hand written sw
ASD:Suite
Create design Models +
interface models in ASD
Generate code
Productivity (code
generation) c.t. trad. dev
-
Testing complete system,
Interaction, data, external
code
Stage 2
Container Terminal (CT)
Verification of the
developed control software
of the CT (generated +
Hand written)
Model based Testing
Spec Explorer
Create Test Model in Spec
Explorer, generate test
suite
Test Productivity
Modeling skills,
complexity costs
Stage 3
Container Terminal (CT)
Verification of the developed control
software of the CT (generated + Hand
written)
Model based Testing
ASDspec & Spec Explorer
Generate Test Model from existing ASD
interface Model , complete Test Model ,
generate test suite
+ Productivity (code generation, partial test
generation)
Testing complete system, Interaction, data,
external code
(benefits only in case of existing of ASD
Models)
CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 16
ASD (ANALYTICAL SOFTWARE DESIGN)
• Model Driven Engineering (code generation)
• Component Based Development
• Models are verified mathematically at design
time (formal methods)
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 17
COMPONENT BASED DEVELOPMENT ASD-STYLE
Code Generation 2014 18
Client
A
B C
Interface A
Interface B Interface C
Client
component
ASD
component
Used
component
Implemented
service
Used
service
Component: Decomposition
Specification
Design
Verification
Generation
Interface model
specifies
behaviour
Design model
implements
behaviour
For each component a interface mode and design model are created in ASD
ASD: WORKFLOW
• Designer defines behavior in component models
• ASD:suite verifies models using model checking
• ASD:suite generates implementation code
19 Code Generation 2014
But some shortcoming
• No support for
Legacy/External code
• No support for indirect
component interaction
• Limited data
interaction
Testing is needed.
RESULTS: ASD-CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 20
Higher overall productivity compared to traditional development.
Main remaining problems:
- Testing interaction and complete system.
- Debugging of third party library (some bugs found in manual written
code, legacy code)
Productivity
CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 21
MBT WITH SPEC EXPLORER
Explore &
Analyze
Remodel
Generate
C# Model (or other .Net
Language)
Model Graph
Test Suite Execute
Nunit
http://www.nunit.com/
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 22
MODELING IN SPEC EXPLORER [TypeBinding("CraneComponent")]
class Crane
{
public Cranestates Cranestatevar;
internal ICrane_NIModel ICrane_NIimpl;
internal ICrane ICraneimpl;
[Rule(Action = "new CraneComponent(craneNr)",
ModeTransition = "crane->crane0")]
Crane(int craneNr)
{
ICraneimpl = new ICraneCranecomponent(this);
craneList.Add(this);
}
State declaration
Initialization
Used components
Spec Explorer
represents models as
annotated C# code Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 23
MODELING IN SPEC EXPLORER
[Rule(Action = "this.PlaceContainer(height)", ModeTransition = "crane4->crane4")]
public void ICrane_PlaceContainer(GripperHeightEnum height)
{
switch (basecomponent.Cranestatevar) {
case Cranestates.Positioned: {
basecomponent.Cranestatevar
= Cranestates.PlacingContainer;
basecomponent.ICrane_NIimpl.PlaceContainer(height);
return;
}
default:{
Condition.IsTrue(false);
throw new InvalidOperationException();
}
};
}
Every model method
represents an action, its
conditions and its effects.
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 24
MBT-VISUALIZATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Completed models can
be explored to
discover all possible
behaviour
25
MBT- VISUALIZATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Explored model
contains all possible
action sequences
26
TEST GENERATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Spec Explorer
creates test
sequences to cover
explored behaviour
27
GENERATED TEST SCRIPTS
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Nunit test scripts are
used to implement
and execute tests
28
RESULTS: SPEC EXPLORER-CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 29
Significantly less effort than traditional automated testing
• Model Based Testing + Software Analysis
• Support of data combination testing
• Support of Model composition, incremental
However some remaining problems:
• Modeling still comparatively expensive
• Modeling effort, complexity and skills (experienced tester needed)
HOW TO COMBINE BOTH BENEFITS
ASD generates verified
software components
Spec Explorer generates
automated software tests
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Complications
• Modeling needed
• MBT skills needed
Complications
• Legacy code
• External code
• Component
Interaction
• Limited support of
data Client
A
B C
Test
script
A
B C
Test
model
30
CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 31
OUR SOLUTION: ASDSPEC
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
ASDSpec generates Test Model (push button) from existing ASD
interface models
Tester need to refine manually the generated model (add behavior,
data, slicing,…)
32
COMBINING THE BEST OF BOTH WORLDS
Automatic generation of partial Spec Explorer test model from existing ASD model
(push button)
Benefits
• Effortless model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
Results:
– High Quality, Reduced cost
– Adds dynamic testing to ASD
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Statistical Dynamic
ASDspec
33
ASDSPEC: WORKFLOW
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Reuse existing ASD interface models to generate
Spec Explorer MBT models automatically
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 34
ASDSPEC: DATA AND MODEL COMPOSITION
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Add data
interaction
Add model
composition
Generated code can be extended with any Spec
Explorer feature
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 35
GENERAL MDSD ARCHITECTURE
Model
Reverse
Engineering
Model
Repository
Model Code,
Doc, etc.
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 37
ASDSPEC ARCHITECTURE
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
ASDSpec
makes use of
existing MDSD
architecture
Model
Reverse
Engineering
Model
Repository
Spec
Explorer
Model
SpecExplorer
Cord
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
ASD
Interface
models
38
ASDSPEC: TECHNOLOGIES
QVTO ASD
XML
SE
model
Cord
script
Acceleo
Acceleo
ASD
EMF
SE
EMF
Cord
EMF
QVTO
QVTO
ASDSpec
Model transformations bridge gap between ASD
and Spec Explorer
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 39
RESULTS: ASDSPEC- CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 40
Spec Explorer ASD + ASDSpec
Approach Generate test suite Generate test model
Techniques MBT MBT
Effort/complexity Medium Low
Test cases 89 93
Perceived effectiveness Medium High
Bugs Some bugs in hand written code
/HAL
All known bus found
• Low effort model creation
- Easy reuse of existing work
• Testing of complete system
- Legacy code, data interaction
• High Quality, Cost Reduction
TRADITIONAL SW DEVELOPMENT TRADITIONAL MANUAL TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Hand written code
Detail design
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Manual
Test suite
41
Summary
TRADITIONAL SW DEVELOPMENT TRADITIONAL AUTOMATED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Hand written code
Detail design
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Test
executi
on
Tool
Manual
scripted
Test suite
42
Summary
ASD BASED SW DEVELOPMENT TRADITIONAL AUTOMATED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated code Hand written code
Interface model
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Manual
scripted
Test suite
Test
executi
on
Tool
43
Summary
ASD BASED SW DEVELOPMENT MODEL BASED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated code
Manual Test
model
Interface model
External code
Interface spec
Config,
data
Legacy code
Integration testing
System testing
Unit testing
Generated
Test suite
Manual
scripted
Test suite
Test
executi
on
Tool
MBT
tool
Hand written code
44
Summary
ASD BASED SW DEVELOPMENT MODEL BASED TESTING+ASDSPEC
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated code
Generated
Test model
Interface model
External code
Interface spec
Config,
data
Legacy code
Integration testing
System testing
Unit testing
Generated
Test suite
Manual
scripted
Test suite
Test
executi
on
Tool
MBT
tool
Hand written code
45
Summary
TOOL STATUS
• Tool currently in prototype phase
• as an Eclipse plugin
• For now available on request
• Future steps
– Support data handling/configuration aspects in
ASDSpec, instead of relying on manual additions in
Spec Explorer,
– Supporting other MBT tools.
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 46
CONCLUSIONS
• MDE and MBT technologies have matured a lot in the
latest years
• It is a matter of time….. Evolution…..
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 47
Using Formal Specifications to
Support Model Based Testing
ASDSpec: A Tool Combining the
Best of Two Techniques
ASD
Static Dynamic
ASDspec
Static Dynamic
ASDspec
top related