hands-on experience model based testing with spec explorer
TRANSCRIPT
MODEL BASED TESTING WITH SPEC EXPLORER By Rachid Kherrazi, Kyra Hameleers and Adrian Iankov
Radboud University Nijmegen 04-12-2015
MODEL BASED TESTING WITH SPEC EXPLORER AND ASDSPEC • Part 1 – Introduction to Model Based Testing
– What is MBT
– Why MBT
• Part 2 – MBT with Spec Explorer
– Modeling, test case generation and execution
– Say Hallo example
Rachid Kherrazi
Integration Design
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
TEST PROCESS
3 main steps in test process
Module
Subsystem
System
3 Rachid Kherrazi
MBT IS THE AUTOMATION OF TEST CASE GENERATION
Manual Automatic
Test case specification (design of logical test cases)
Test case execution (and reporting)
Test case generation (design of physical test case)
(Actions & expected results)
Model base
d t
est
ing
Tra
dit
ional m
anual te
stin
g
Tra
dit
ional auto
mati
c t
est
ing
Manual C# Scripting
Testing skills + Scripting skills + Modeling skills
Modeling
Lightweight
Modeling languages
Generate
Scripting
Rachid Kherrazi
MBT PROCESS IN A NUTSHELL
Requirements
SUT
Expected outputs
(Test Oracle)
Inputs
(Test sequences)
Verdict Failed Actual result # expected
result
Test suite
generate
author
feedback
feedback
feedback
Feedback
Model based analysis
(via e.g. model exploration, Model checking)
control
Model
Actual outputs
Implementation Bug
Modeling error
Specification error
Design error
Conformance failures
Rachid Kherrazi
TYPICAL ISSUES WITH TRADITIONAL TESTING
• Many problems discovered at a late stage of development process, often costly to resolve.
• In addition, they affect the planning of development processes in that they must be resolved 'In between'
Several pre and post release defects not found with traditional testing, which results in
Several delays in planning
With related quality/cost impact
Typical what we
hear from our
customers
Rachid Kherrazi
ROOT CAUSE
• Unit testing is insufficient
– Single unit may work properly in isolation
– Incorrect interaction between units may cause serious reliability failures
• System-level testing
– Requires model of system behavior
– Behavior is often reactive/nondeterministic
• Implementation is multi-threaded or distributed
• Thread scheduling hard to control
– State space is typically infinite
– Traditional FSM-based testing does not scale
Rachid Kherrazi
MBT HELPS TO AVOID INTEGRATION ISSUES AND POST RELEASE DEFECTS
Review
Static testing
Unit testing
Manual testing
happy flows
Manual testing
alternative flows
Manual testing
bad weather flows
Typical uncovered
Post release defects
Typical uncovered
Integration issues
This is typical domain
where Model Based
Testing helps
You can try to avoid
integration issues and
post release defects by
adding manual test case,
but it will be complex
will take lot of effort and
finally costly
Increased complexity and effort of testing
Manual exhaustive
exploratory testing
Rachid Kherrazi
MODEL BASED ANALYSIS AND TESTING
Unit testing
Integration testing
System testing
Acceptance testing
Coding
Design
Specification
Requirements
Static analysis
Dynamic Analysis
Model based SW Analysis Visualization, simulation, safety
analysis (design errors)
Model based SW testing Execution of test suite against
implementation
Some MBT tools (like NModel, PyModel, Spex Explorer, ModelJUnit) allow also Model Based Analysis
e.g. safety violations (where the program reaches forbidden states), deadlocks (where the program seems to stop running
and stops responding to events) livelocks (where the program cycles endlessly but can’t make progress)
Simulation= A model-based analysis technique, where runs of a model program are observed
Rachid Kherrazi
OTHER DRIVERS FOR MBT
Our world evolves….and becomes complexer
– Cyber-Physical Systems/EoT
– Machine2Machine
– Digital Transformation
Increased need to manage complexity
Increased need to technology enablers
Innovative company’s need innovative tools and approaches to develop innovative products and services
Model Based System Engineering
Model Driven Engineering
Model Based Testing
Rachid Kherrazi
MBT TOOLS AND SPEC EXPLORER FAMILY
• Explosion of tools (most from academia, little products)
• Trend
– from graphical to textual modeling
– Platform independent /open sources
• The story of the Spec Explorer “family”
ASML (Microsoft research) NModel
PyModel
ModelJUinit
RubyModel
…..
Open sources
Textual modeling
Spec Explorer
Add-in MS VS (Product, support,
documentation)
History
Platform independent
Rachid Kherrazi
MODELING WITH SPEC EXPLORER: LOOK AND FEEL
Explore &
Analyze
Remodel
Generate
C# Model (or other .Net
Language)
Model Graph
Test Suite Execute
Nunit
http://www.nunit.com/
Rachid Kherrazi
Environment
System Under Test
E
S
Abstraction
SE
Model based
testing
S: includes all details of SUT but nothing about expected environment
E: full info about environment but nothing about SUT behavior
SE: most extreme situation, too match detail, unpractical
Abstraction is essential M is typical for model based testing
Abstraction: data, function, communication abstraction…
M
MODEL ABSTRACTION
Rachid Kherrazi
EXAMPLE: HELLO BEST TESTERS
A typical use case:
# Action Expected result
1 Start application App. started
2 Press Say Hello button Text box = “Hello Best Testers”
3 Press Clear button Text box = “ “
4 Exit application App. closed
Rachid Kherrazi
MODEL: A SIMPLE FSM
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
States, Variables, Actions, pre and post condition
Rachid Kherrazi
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Random walk with length N=10
TEST SUITE GENERATION
Rachid Kherrazi
TEST SUITE GENERATION
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Visit all state
Rachid Kherrazi
TEST SUITE GENERATION
Not Running
<TextBox> Running
<TextBox=“ “>
Running
<TextBox = Hello Best
Testers>
Clear
Clear
Say Hello
Say Hello
Exit
Start
Exit
Test selection criteria: Chinese Postman (visit all transitions a.s.a.p) (shortest path)
Rachid Kherrazi
C# MODEL namespace DemoModel
{
public enum State
{
NotRunning,
Running
}
static class ModelProgram
{
static State appStatus = State.NotRunning;
static string TextBox=" ";
[Probe]
public static string Satatus()
{
return string.Format("appStatus:{0},TextBox:{1}", appStatus, TextBox);
}
[AcceptingStateCondition]
static bool InitialStatus()
{
return appStatus == State.NotRunning ;//&& TextBox == " ";
}
States declaration
Initialization
Optional, just for visualization in graph
When to stop in case of test case
Rachid Kherrazi
C# MODEL
[Rule(Action = "StartApp()/result")]
static Implementation.State StartApp()
{
Condition.IsTrue(appStatus == State.NotRunning);
Requirement.Capture("requirement1.1:StartApp");
appStatus = State.Running;
return (Implementation.State)appStatus;
}
[Rule(Action = "StopApp()/result")]
static Implementation.State StopApp()
{
Condition.IsTrue(appStatus == State.Running);
Requirement.Capture("requirement1.2:StopApp");
appStatus = State.NotRunning;
TextBox = " ";
return (Implementation.State)appStatus;
}
}
}
Rachid Kherrazi
C# MODEL
[Rule(Action = "ClearText()/result")]
static string ClearText()
{
Condition.IsTrue(appStatus == State.Running);
TextBox = "";
Requirement.Capture("requirement1.3:ClearText()/result");
Console.WriteLine("Clear text");
return TextBox;
}
[Rule(Action = "SayHallo(A)/result")]
static string SayHallo(string A)
{
Condition.IsTrue(appStatus == State.Running);
Requirement.Capture("requirement1.4:SayHallo");
TextBox = A;
return TextBox;
}
Action declaration
Requirement capture
Rachid Kherrazi
Offline
Tests are generated and stored in files
Test execution is separate (different environment)
Can generate ones, but executed many times
e.g.VS or Nunit, QulaityCenter
MBT Tool
Offline testing
Online
Tests are executed in parallel with generation
MBT tool play the game with system under test
Good for testing non-deterministic systems
MBT Tool
Online testing
MBT is…..
Strategies, tools and artifacts
Manage complexity
Reduces the need of manual or human involvement of interaction
Avoids spending time in unskilled repetitive error prone or redundant tasks
Provides bandwidth to Innovate!!