cucumber spec - a tool takes your bdd to the next level
TRANSCRIPT
PROJECT TEAM
2
• Adrian Yankov
– Graduation Intern
• Kyra Hameleers
– Test Engineer @ Nspyre
• Rachid Kherrazi
– MBT competence Leader @ Nspyre
– Project Leader of this project
GOALS
• Sharing knowledge on 3 topics
– Behaviour-Driven Testing
– Model-Based Testing
– Combined strengths
• Encourage the usage of Model-Based Testing
CONTENT
• Introduction
– Behaviour-Driven Testing(BDT)
• What is it and why?
– Model-Based Testing(MBT)
• What is it and why?
– Using Gherkin in MBT
• CucumberSpec Tool
• Testing a drone with this approach
4
BDT LANGUAGE AND BENEFITS
Why?
• Main benefit – the team gets together and writes down a
common, shared knowledge in a well-known format.
• Generates test cases from a high-level formal human
language.
Tools
• There are several tools such as SpecFlow, Behave and
Cucumber etc.
6
SOME NICE FEATURES
8
• Data Parameterization
– Supports placeholders.
– Contained within < >
Example:
BDT WITH SPECFLOW
9
GUI
Application/
database layer
Test
inte
rface Physical test
cases
Logical test
cases (Gherk
in)
Requirements
Features
Use cases
Test Engineer Architect/
designer
SUT
create
MODEL-BASED TESTING
Manual Automatic
Testcase specification (design van logische test cases)
Testcase execution (includes reports)
Testcase generation
Model
base
d t
est
ing
Tra
dit
ional
manual
test
ing
Tra
dit
ional
auto
mati
c t
est
ing
Manual C# Scripting
Testing skills + Scripting skills + Modeling skills
Modeling
Lightweight
Modeling languages
Generate
Scripting
10
MBT WITH SPEC EXPLORER
12
GUI
Application/
database layer
Test
inte
rface
Automatic
generation
of test
scripts
Test Model
Model includes
expected
behavior
Requirements
Features
Use cases
Gherkin
Test Engineer Architect/
designer
SUT
create create
BDT IN COMBINATION WITH MBT
Our tool CucumberSpec generates a SpecExplorer test
model from the separate scenarios.
13
CucumberSpec Isolated scenarios
USING GHERKIN IN MBT
Benefits
Visualization of interaction space, generate more and better (exhaustive) testcases
by using different smart exploration strategies to suit the tester’s needs.
14
GUI
Application/
database layer
Test
inte
rface
Automatic
generation
of test
scripts
Test Model
Model includes
expected
behavior
Requirements
Features
Use cases
Gherkin
Test Engineer Architect/
designer
SUT
CucumberSpec
check create
CUCUMBERSPEC SUMMARY
Idea of our CucumberSpec
• Generates a Test model from a feature file, Connecting the separate scenarios in a model
• Intention is to support most features (parameterization, tags….)supported by BDD tools like Behave and SpecFLow
• + A lot of “Model Based” Benefits,
• Generate more tests cases based on different coverage used in MBT
– Testing interaction between the scenarios
– More data permutations and combinations
– This allows exhaustive testing
• No refinement is needed.
• Fully automatic process.
16
AR.DRONE
18
Mobile AR.Drone
SW Hardware
Router Camera 1 Motors
Camera 2 Hoogte
Sensor GPS
Application
UI
Controller
Communication
Business Layer
Data Acces Layer
Commun
ication
Layer
REQUIREMENTS
• Functional requirements:
1. Takeoff and Land via Android commands
2. Fly around via Android commands
3. Always be able to use the emergency command
• Non-Functional requirements:
1. Usability
2. Reliability
3. Security
19
To keep it simple in this PoC, we
focused only on a limited
number of functional
requirements.
TEST SCOPE
• Functional requirements
1. Takeoff and Land via Android commands
1. Takeoff
2. Land
2. Fly around via Android commands
1. Left/Right
2. Up/Down
3. Forward/Backward
4. Turn Lef/right
3. Always be able to use the emergency command
20
TESTING THE DRONE
21
Actions:
1. Take off
2. Emergency
3. Land
4. R: Right
5. L: Left
6. F: Forward
7. B: Backward
8. l: Turn left
9. r: Turn right
10. u: Up
11. d: Down
TEST MODEL States:
1. Take off
2. Flying
3. Landing
4. Landed
5. Emergency
22
Actions:
1. Take off
2. Emergency
3. Land
4. R: Right
5. L: Left
6. F: Forward
7. B: Backward
8. l: Turn left
9. r: Turn right
10. u: Up
11. d: Down
Take off Flying Landing Landed Emergency
Take off x Reached 30 cm land X Emergency
Flying x
Right/ Left/ Up/ Down/
Forward/ Backward/ Turn
left/ Turn right land x Emergency
Landing take off, height<0 take off, height>0 X Reached 0 cm Emergency
Landed take off X X x Emergency
Emergency x Emergency, height>0 X Emergency, height<0 x
Some issues with traditional test design:
Traditional state transition test technique does not scale
high coverage (>1-switch) is complex, costly and difficult to
maintain
CREATE GHERKIN (SMALL EXAMPLE FOR EXPLANATION PURPOSES)
23
0-switch state transition test
cases in a high-level formal
human language(Gherkin).
Landed Flying
Landed x Take off
Flying Land Forward/
Backward
GENERATED SPEC EXPLORER (SMALL EXAMPLE FOR EXPLANATION PURPOSES)
25
Visualization of
interaction space,
This allows generation
of more and better
(exhaustive) testcases
by using different smart
exploration strategies
(supported by the MBT
tool) to suit the tester’s
needs.
BACK TO TEST SCOPE
27
Take off Flying Landing Landed Emergency
Take off x Reached 30 cm land X Emergency
Flying x
Right/ Left/ Up/ Down/
Forward/ Backward/ Turn
left/ Turn right land x Emergency
Landing take off, height<0 take off, height>0 X Reached 0 cm Emergency
Landed take off X X x Emergency
Emergency x Emergency, height>0 X Emergency, height<0 x
TEST EXECUTION
• Test code can be generated
• The Tool can generate test
scripts for different execution
tools e.g NUnit/JUnit.
• No code to test it too
31
CONCLUSION
33
Metrices Cucumber MBT CucumberSpec
Approach
using gherkin files and action
definition to generate and
execute test suite
using models to
generate and execute
test suite
using gherkin files to generate
models, and generate and
execute test suite
Technique Behavior Driven Testing Model Based Testing
Combining strength of BDD and
MBT
Modeling Create gherkin files Create test model Generate test model
Effort High Medium/High Medium
Coverage 0-switch Transition coverage Transition coverage
Number of testcases 20 7 7
NEXT STEP • Next step: Explore automatic test execution via
Xamarin/Calabash (benefit cross-platform testing)
• We are looking for a pilot project based on real industry case from industry
• If you are interested please contact us
• or send a mail to [email protected]
34
SUMMARY
• Combining the best of 2 world
– Behavior-Driven Testing
– Model-Based Testing
35
CucumberSpec Isolated scenarios