practical use of model-based testing in the agile ...not only finite state machine models are great...
TRANSCRIPT
Practical use of Model-Based Practical use of Model-Based Testing in the Agile Development Testing in the Agile Development
CycleCycle
Kristian Karl
Copyright Prolore AB. All rights reserved.
Kristian Karl
Test consultant at Prolore, StockholmTest automation since 1994Model-based testing since 2004Founder of mbt.tigris.org
2
Agenda
3
What's different with mbt.tigris.org
How to model?
Metrics, and how to present results
Model-based testing and Agile Development
Showcases and key success factors
What's different with mbt.tigris.org
4
What's different with mbt.tigris.org
5
No UMLNo [graphical] exit/stop pointsFocus on OnlineConnects to any tool that understands Java or SOAP
No UML
6
A shallower learning curve• mbt.tigris.org's simple rule-set, in conjunction
with GraphML, makes it easier for tester to get started.
• As testers, we do not need all functionality that UML has to offer.
Typically, a tester can start designing after only a few hours.
No [graphical] exit/stop points
7
The way to tell mbt.tigris.org to stop generating a test sequence, are done by means of Stop Criteria.
Stop Criteria are passed as arguments to the tool, and they can be combined.
Creates different types of tests from the same model.
• Functional tests• Smoke tests• Bug hunting tests
Stop Criteria
8
REACHED_EDGE
REACHED_VERTEX
EDGE_COVERAGE
VERTEX_COVERAGE
TEST_LENGTH
TEST_DURATION
REACHED_REQUIREMENT
REQUIEMENT_COVERAGE
NEVER
Online
9
Supports online test sequence generation.Bug hunting! Generate long and unpredictable test sequences.
• We do not want to walk the same path every time we execute an automated test.
• We want variation, spiced with randomness. This will actually find bugs, not only verify requirements!
Enables non-deterministic tests.
Running Online
10
Instantiated in a Java test:org.tigris.mbt.ModelBasedTesting.getInstance()
SOAP – Running mbt as a web service● SetCurrentVertex● GetDataValue● ExecAction● GetNextStep● GetStatistics
● HasNextStep● Load● Reload● PassRequirement
The Walks
11
RANDOMA_STARSHORTEST_NON_OPTIMIZED
RANDOM
12
Navigates through the model in a completely random manor. Also called "The Drunkard’s walk". The algorithm selects an out-edge from current vertex by random, and repeats the process in the next vertex. Example:
java -jar mbt.jar offline -g RANDOM -s EDGE_COVERAGE:100 -f UC01.graphml
A_STAR
13
Will try to generate the shortest possible test sequence through a model with given stop condition. The algorithm only works well on smaller models.
Example:
java -jar mbt.jar offline -g A_STAR -s EDGE_COVERAGE:100 -f UC01.graphml
SHORTEST_NON_OPTIMIZED
14
This is a compromise between A_STAR and RANDOM. The algorithm works as follows:
1. Choose an edge not yet visited by random.
2. Select the shortest path to that edge using Dijkstra's algorithm.
3. Walk that path, and mark all walked edges as visited.
4. When reaching the chosen edge in step 1, start all over, repeating steps 1 → 4.
Example:
java -jar mbt.jar offline -g SHORTEST_NON_OPTIMIZED -s EDGE_COVERAGE:100 -f UC01.graphml
Command options in xml
15
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE MBTINIT SYSTEM "mbt_setup.dtd" ><MBTINIT EXTENDED="true" EXECUTOR="java:myPackage.myTestClass" LOG-COVERAGE="10"> <MODEL PATH="model/Main.graphml"/> <MODEL PATH="model/basicSubModels"/> <SCRIPT PATH="model/basicFunctions.bsh"/> <SCRIPT> x = 0; </SCRIPT> <GENERATOR TYPE="A_STAR"> <CONDITION TYPE="REACHED_VERTEX" VALUE="v_MainWindow_DB_Loaded"/> </GENERATOR> <GENERATOR TYPE="RANDOM"> <OR> <AND> <CONDITION TYPE="EDGE_COVERAGE" VALUE="100"/> <CONDITION TYPE="VERTEX_COVERAGE" VALUE="100"/> </AND> <CONDITION TYPE="TEST_DURATION" VALUE="900"/> </OR> </GENERATOR></MBTINIT>
How to model?
16
The editor yEd from yWorks
17
mbt.tigris.org - Syntax
18
Edges
19
An edge is a transition. Could be a click on a button, a timeout, a server API call.
An edge can have a label, but it is not mandatory.• The label of an edge will map against a method, function
or sub routine during test execution.
As a rule of thumb, the best practice is to start the name with 'e_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as an edge. For example: e_StartApplication e_EnterBasicView
The same name of an edge, can be re-used in the model.
Vertices
20
A vertex is some kind of state that is possible to verify using some kind of oracle.
A vertex must always have a label.• The label of a vertex is mapped against a method,
function or sub routine during test execution.
As a rule of thumb, the best practice is to start the name with 'v_'. The reason for this, is that it is easier to recognize the function in the test execution tool, as a vertex. For example: v_ApplicationStarted v_BasicView
The same name of a vertex, can be re-used in the model.
Labels
21
Label [cond.expr.] / Statement1;Statement2;
White spaces not allowed. Follow the naming convention for the implementing programming language.
Example:e_StartApplication
Example:e_EnterBasicView
Guards
22
Label [cond.expr.] / Statement1;Statement2;
Conditional expression that returns a Boolean value.The guard indicates if the edge is accessible in the current state of the machine. The language is either Java or JavaScript.
Example:e_Evaluate [x>0 && y<=10]
Example:e_Checkout [shoppingBasket.contains("Cerials")]
Statements
23
Label [cond.expr.] / Statement1;Statement2;
Any applicable Java or JavaScript statements
Example:e_Calculate / x=5;y=10*x;
Example:e_AddCerials / shoppingBasket.add("Cerials");
Keyword - Start
24
Used by Vertices only.
Each graph must have a start vertex, and that vertex holds this keyword. There is only one vertex in a graph, holding this keyword.The Start vertex can only have one out-edge.If the graph is the main-graph, that edge must have a label.If the graph is a sub-graph, that edge cannot have a label.
Keyword - REQTAG
25
Used by Vertices.Used by MBT for keeping track of which requirements are fulfilled during testing.
In the example above, if v_BookInfomation is passed, it would mean the the requirement with tag UC01 2.2.3 has passed.
Keyword - weight
26
Used by Edges only.
Used by MBT during random walks during test sequence generation. It holds a real value between 0 and 1, and represents the probability that a specific edge should be chosen. A value of 0.05, would mean a 5% chance of that edge to be selected during a run.
Note: It only works with the RANDOM generator, and must be enabled using the command-line switch -w
Keyword - BLOCKED
27
Used by both Vertices and Edges.
A vertex or an edge with the key word BLOCKED, will be excluded from the model when walked.
Keywords for Sub-graphs
28
The Stop vertex denotes an exit point in a sub-graph. Only sub-graphs can have a stop vertex.
MERGE is used when merging graphs. It means that the vertex containing this key word, should be merged with the first occurrence of a vertex with the same label.
NO_MERGE is used when merging graphs. It means that the vertex containing this key word, should not be merge (or expand) this vertex with correspondent sub graph.
Test Data
29
Test Data
30
Test Data
31
Test data should be provided by a tool (written in Java), which acts as an abstraction layer between the test and the data sources:
• tool.GetRandomLoginCredentials()• tool.GetRandomPerson()• tool.GetRandomBook()
Example - EFSM
32
Example – Requirements
33
Example - Sub-graph
34
Main graph
Sub-graph C
Result whenmerged
Metrics, and how to present Metrics, and how to present resultsresults
35
Metrics
36
Standard test metrics does not apply so well.Test cases are not meaningful when running online tests.MBT tends to drive the requirement work.
Metrics
37
Focuses on requirements (or verification points)
• ”How many requirements should have been verified?”or:
• ”How many verification points should have been verified?”
Reporting
38
Requirement Database• Ties requirements to Sprints• Ties requirements to Teams
Test Result Database.• Stores data from every test run• Knows what requirements was verified
Dashboard• Displays what requirements are verified• Allows drill-downs
Reporting - Example
39
Model-based testing and Model-based testing and Agile DevelopmentAgile Development
40
Model-based testing and Agile Development
41
Side-by-side from day one• The testers starts modeling immediately.• It is not unusual that test automation is
ready before developers.
Test is more exposed than earlier• Visualization• More people understands what test is
actually doing.• Test is getting more feedback.
Model-based testing and Agile Development
42
Not only Finite State Machine Models are great in Agile test, also consider Classification Trees Models.
Model-based testing and Agile Development
43
The use of mockups is very important.
• In a sprint, the team may not have all the functionality needed to run the tests. This is solved by using mockups.
The tests must be able to run on different environments.
• Test Data Tool
Version handling in the same branch as Dev.
Model-based testing and Agile Development
44
The Dev teams delivers to the Integration team after the demo.The Integration team has maintenance responsability.The Integration team runs continuous nightly builds, deploys and run all delivered tests.
Showcases and key success Showcases and key success factorsfactors
45
Swedish Pensions Agency
46
3-tier Client / Server• Mysql, Oracle, JBoss, Java, Perl
Mixed new functionality and maintenance of legacy codeTesting interfaces:
• Thin GUI clients• Fat GUI clients• Web Services• Batch jobs
4 dev teams, 1 integration team
Swedish Pensions Agency
47
One team test automated every test.
Test data tool – GTAD [Generate Test Automation Data]
10 testers wrote 69 trouble reports in 8 months.• What if we would have written 1.000 trouble
reports, and spent 2 hours on every report?• Many issues very exposed early because of the
models.
Every night we executed all our automated tests:• Unit tests• Functional API-tests• GUI tests• Performance tests
Neonode - N2
48
Any test should be able to run on any device.Automated GUI testsCall setupsSMS/MMS sending and receiving
Business Values
49
Proven technique for test automation in Agile Dev. Teams.
• Enables QA people with business domain knowledge to design automated tests.
• Models can be re-used for different purposes.
Will find bugs.Long life & Durability
Key Success Factors
50
Start modeling early.
A high abstraction level of the models.
Happy paths.
Show the models!
Use the tests for bug hunting, not just verification of requirements.
• This means online tests.
Use mockups.
Test Data Tool
Links
51
The tools:http://mbt.tigris.orghttp://www.yworks.comhttp://www.berner-mattner.com/en/berner-mattner-home/products/cte-xl/index.html
Email:[email protected]@gmail.com