about the presenter€¦ · model based generation of test cases tutorial antti huima managing...
TRANSCRIPT
![Page 1: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/1.jpg)
1
Copyright © Conformiq Software Ltd. All rights reserved.
www.conformiq.com
Model Based Generation of
Test CasesTutorial
Antti HuimaManaging Director
Conformiq Software Ltd
Copyright © Conformiq Software Ltd. All rights reserved. 2
Model Driven Quality Assurance
About the PresenterAntti Huima (M.Sc. Tech.)• Managing director of Conformiq
Software Ltd. (formerly R&D director)Conformiq Software Ltd.• Est. 1998• Model driven testing and quality
assurance
![Page 2: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/2.jpg)
2
Copyright © Conformiq Software Ltd. All rights reserved. 3
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
Copyright © Conformiq Software Ltd. All rights reserved. 4
Model Driven Quality Assurance
The TTCN-3 Platform• Expressing testing logic and data• Exchanging and sharing tests• Executing tests automatically
![Page 3: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/3.jpg)
3
Copyright © Conformiq Software Ltd. All rights reserved. 5
Model Driven Quality Assurance
The Problem of Test Design• How to design tests for the TTCN-3
platform?• Manual test case design takes time...• ...and creates risks!
– Missing test cases– Invalid test cases– Redundant test cases
• 20-50% test cases in the telco segment are invalid (buggy) before the first run against the system under test
Copyright © Conformiq Software Ltd. All rights reserved. 6
Model Driven Quality Assurance
Missing Test Cases• A human engineer can accidentally
miss a test case that is dictated by requirements, e.g. for:– An error handling case– A limit value of a data parameter– The expiration of a rarely activated timer
Required test cases
Missed test case(not in test set)
![Page 4: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/4.jpg)
4
Copyright © Conformiq Software Ltd. All rights reserved. 7
Model Driven Quality Assurance
Invalid Test Cases• “Invalid test case” = yields sometimes
wrong verdict– PASS when observed execution is not correct
w.r.t. requirements– FAIL when observed execution is correct w.r.t.
requirements
Required test cases
Invalid test case(yields wrong verdict)
Copyright © Conformiq Software Ltd. All rights reserved. 8
Model Driven Quality Assurance
Redundant Test Cases• Redundant cases increase the
complexity of test suites and the maintenance costs– Duplicate test cases– Requirements covered in other test cases– Test cases outside the specification
Required test cases
Duplicate test cases
Test case outside spec
![Page 5: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/5.jpg)
5
Copyright © Conformiq Software Ltd. All rights reserved. 9
Model Driven Quality Assurance
A Bottom Line
Net value=Other maintenance costs-Risks and costs of invalid test cases-Risks of missing test cases-Costs of redundant test cases-Direct development costs-Intrinsic value of a good test suite
Copyright © Conformiq Software Ltd. All rights reserved. 10
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 6: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/6.jpg)
6
Copyright © Conformiq Software Ltd. All rights reserved. 11
Model Driven Quality Assurance
Model Driven Testing• Model driven testing is a solution to
the problem of designing and maintaining tests
• Also known as– model-based testing– specification-based testing– specification driven testing
• It complements solutions for– test management– test execution (e.g. TTCN-3)
Copyright © Conformiq Software Ltd. All rights reserved. 12
Model Driven Quality Assurance
Logic of MDT• The model describes the expected
behavior of the SUT as an open system
• The MDT tool synthesizes an environment that drives the real SUT in order to check that it works as the model predicts
![Page 7: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/7.jpg)
7
Copyright © Conformiq Software Ltd. All rights reserved. 13
Model Driven Quality Assurance
Heuristics• Model driven testing heuristics are
used to make sure that all important functionality of the model is exercised, for example:– statechart state and transition coverage– date definition / use coverage– boundary value analysis– condition and atomic condition coverage– requirements coverage
Copyright © Conformiq Software Ltd. All rights reserved. 14
Model Driven Quality Assurance
Reference Implementation• The model is basically an abstract
reference implementation of the SUT, because it– is executable– describes the behavior of the SUT (albeit
generally on a higher level of abstraction)
• As a matter of fact, the SUT model can be often simulated
![Page 8: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/8.jpg)
8
Copyright © Conformiq Software Ltd. All rights reserved. 15
Model Driven Quality Assurance
“The MDT Axiom”
Testsgenerated from a model
never failwhen executedagainst a run ofthe same model
Copyright © Conformiq Software Ltd. All rights reserved. 16
Model Driven Quality Assurance
MDT Process1. Identify system under test (SUT)2. Create model of the SUT’s expected
behavior
• Online approach no intermediatetest scripts—not covered in this talk
• Offline approach generateintermediate test scripts
![Page 9: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/9.jpg)
9
Copyright © Conformiq Software Ltd. All rights reserved. 17
Model Driven Quality Assurance
Offline TTCN-3 Approach3. Command the MDT tool to generate
test cases in TTCN-34. Store the generated test cases in
version control / configurationmanagement system (optional step)
5. Execute the generated tests on a separate TTCN-3 platform (e.g. company internal tools, or fromTelelogic, OpenTTCN, Danet, Elvior, TestingTech...)
Copyright © Conformiq Software Ltd. All rights reserved. 18
Model Driven Quality Assurance
Offline TTCN-3 Approach
Model
MDT Tool
TTCN-3 Scripts TTCN-3 ExecutionPlatform IUT
Test Harness
Reports
TRI & TCI
![Page 10: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/10.jpg)
10
Copyright © Conformiq Software Ltd. All rights reserved. 19
Model Driven Quality Assurance
The State-of-the-Art Solution• Derive test cases automatically from
functional models...• generating also test data, time, and
expected results (test oracles)...• using well-established heuristics like
model-level branch coverage or boundary value analysis...
• and algorithms including symbolic state space analysis, constraint solving and combinatorial optimization
Copyright © Conformiq Software Ltd. All rights reserved. 20
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 11: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/11.jpg)
11
Copyright © Conformiq Software Ltd. All rights reserved. 21
Model Driven Quality Assurance
Basic WorkflowUpdate Model
Analyze in MDT tool
GenerateTTCN-3
Compile Execute againstSUT
AnalyzeResults
Update humanengineered
TTCN-3 parts
Update SUT
Copyright © Conformiq Software Ltd. All rights reserved. 22
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 12: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/12.jpg)
12
Copyright © Conformiq Software Ltd. All rights reserved. 23
Model Driven Quality Assurance
• Our commercial tool for model driven testing
• Open architecture• Windows, Linux
TM
Copyright © Conformiq Software Ltd. All rights reserved. 24
Model Driven Quality Assurance
Offline TTCN-3 Approach (Qtronic)Java/C#
+ statecharts
Qtronic
TTCN-3 Scripts
TTCN-3 ExecutionPlatform IUT
Test Harness
Reports
TRI & TCI
TTCN-3 plugin
Open API
![Page 13: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/13.jpg)
13
Copyright © Conformiq Software Ltd. All rights reserved. 25
Model Driven Quality Assurance
Models• Extended Java• Optional UML state charts
• Created in– Any text editor (Java parts)– Qtronic Modeler (UML state charts)– Third party UML tools (e.g. IBM Rational, Mentor
Graphics tools, Enterprise Architect)
Copyright © Conformiq Software Ltd. All rights reserved. 26
Model Driven Quality Assurance
Supported Constructs• Full data (strings, numbers, records, classes,
arrays…)
• Full time (timeouts, dynamic timeouts…)
• Full control structures (methods, dynamic polymorphism…)
• Full concurrency (multiple Java threads in model, ITC primitives…)
• Full Java + templates, macros, record values, type inference…
![Page 14: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/14.jpg)
14
Copyright © Conformiq Software Ltd. All rights reserved. 27
Model Driven Quality Assurance
Simple Model Anatomy• system block declares ports of the SUT
– These map to the ports of the MTC
• record declarations describe message types– These map to type records in TTCN-3
• class declarations describe active objects– Behavior described in state charts and/or Java/C#
syntax– These are not mapped to TTCN-3 because they are
used to explain behavior
• void main() serves as the entry point
Copyright © Conformiq Software Ltd. All rights reserved. 28
Model Driven Quality Assurance
System Block / MTC ComponentQtronic
system {Inbound moneyIn : Coin;Inbound keyPress : SelectCola,
CancelPurchase;Outbound moneyOut : Coin;Outbound product : Cola;
}
TTCN-3
type port keyPressPort message {out SelectCola;out CancelPurchase; }
type port moneyInPort message {out Coin; }
type port moneyOutPort message {in Coin; }
type port productPort message {in Cola; }
type component SystemType {port keyPressPort keyPress;port moneyInPort moneyIn;port moneyOutPort moneyOut;port productPort product; ... }
![Page 15: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/15.jpg)
15
Copyright © Conformiq Software Ltd. All rights reserved. 29
Model Driven Quality Assurance
Records / Record TypesQtronic
record CancelPurchase { }record Coin { int valueCents; }record Cola { }record SelectCola { }
TTCN-3
type record CancelPurchase { }type record Coin { integer valueCents }type record Cola { }type record SelectCola { }
Copyright © Conformiq Software Ltd. All rights reserved. 30
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 16: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/16.jpg)
16
Copyright © Conformiq Software Ltd. All rights reserved. 31
Model Driven Quality Assurance
Integrating Generated and Manually Written TTCN-31. “Independent PTC” approach
• E.g. computer generated interaction at one interface, manually generated interaction at another one, interactions mostly independent
2. “Filtering” approach• Computer generated behavior is modified and
extended dynamically, e.g. by transforming data structures
3. “Framework” approach• Generated code uses hand-written infrastructure
libraries, e.g. codec-like libraries written in TTCN-3
Copyright © Conformiq Software Ltd. All rights reserved. 32
Model Driven Quality Assurance
Independent PTC Approach
MTCPTC
SUT
Computer generated
![Page 17: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/17.jpg)
17
Copyright © Conformiq Software Ltd. All rights reserved. 33
Model Driven Quality Assurance
Filtering Approach
MTCPTC
SUT
Computer generated
Copyright © Conformiq Software Ltd. All rights reserved. 34
Model Driven Quality Assurance
Framework Approach
MTC
Infrastructure library
SUT
Computer generated
![Page 18: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/18.jpg)
18
Copyright © Conformiq Software Ltd. All rights reserved. 35
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
Copyright © Conformiq Software Ltd. All rights reserved. 36
Model Driven Quality Assurance
How Qtronic Selects Tests• Qtronic uses “symbolic execution” to
“simulate” the system model• It “invents” the environment for the
system using “constraint solving”• The environment is constructed so
that it drives the system model to interesting cases, e.g. boundary cases for parameters
• The intelligent simulations are than mapped to test cases
![Page 19: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/19.jpg)
19
Copyright © Conformiq Software Ltd. All rights reserved. 37
Model Driven Quality Assurance
Coverage Criteria• Transition coverage
– Cover all transitions in all state charts• State coverage
– Cover all states in all state charts• Branch coverage
– For every if and while loop, cover both the positive and the negative branch
• Condition coverage– For every x and y and x or y, cover combinations of the truth
values of x and y (but taking short-circuited evaluation into account)
• Requirements coverage– For every requirement link in the model, cover the link
• Boundary value pattern– For every test x < y, cover cases x = y – 1; x < y – 1; x = y;
and x > y– Other comparators work analogously
Copyright © Conformiq Software Ltd. All rights reserved. 38
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 20: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/20.jpg)
20
Copyright © Conformiq Software Ltd. All rights reserved. 39
Model Driven Quality Assurance
Reporting and Traceability• Qtronic provides information about
the system-level requirements and model-level constructs exercised by test cases
• This information can be exported out of test runs e.g. with log
Copyright © Conformiq Software Ltd. All rights reserved. 40
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 21: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/21.jpg)
21
Copyright © Conformiq Software Ltd. All rights reserved. 41
Model Driven Quality Assurance
Tool Support & Availability• Conformiq Qtronic is the best
available solution for model driven test generation for TTCN-3
• Our competitors include– T-Vec– Leirios– SpecExplorer from MSR– ATG for I-Logix Rhapsody– Reactis– Academic tools
• Visit our stand at this conference!
Copyright © Conformiq Software Ltd. All rights reserved. 42
Model Driven Quality Assurance
Contents
Conformiq Qtronic
DemonstrationsTool Support and AvailabilityReporting and TraceabilityHow Tests are SelectedIntegrating Generated and Manually Designed TTCN-3
Basic Workflow for MDT with TTCN-3Model Driven TestingTTCN-3
![Page 22: About the Presenter€¦ · Model Based Generation of Test Cases Tutorial Antti Huima Managing Director ... • Model driven testing heuristics are used to make sure that all important](https://reader034.vdocuments.mx/reader034/viewer/2022042310/5ed7d4ae8e533201dc4ae5e2/html5/thumbnails/22.jpg)
22
Copyright © Conformiq Software Ltd. All rights reserved. 43
Model Driven Quality Assurance
Conformiq Software Ltd.Innopoli 1Tekniikantie 12FI-02120 EspooFINLAND
Tel +358 10 286 6300Fax +358 10 286 6309
QUESTIONS, COMMENTS?
Antti HuimaManaging DirectorEmail: [email protected]: +358 40 528 8667