seminar: software quality and safety learning by testing
DESCRIPTION
Seminar: Software Quality and Safety Learning By Testing. Volker Fortströer. Motivation [1]. How improve quality control for reactive systems? Availability of a specification of system behavior Rarely precise & reliable because of last minute changes etc. - PowerPoint PPT PresentationTRANSCRIPT
Seminar: Software Quality and Safety
Learning By TestingVolker Fortströer
Volker Fortströer – Learning by Testing 2
How improve quality control for reactive systems?
Availability of a specification of system behavior◦ Rarely precise & reliable because of last minute
changes etc. But a specification of the system is needed to
test and to verify the behavior Automatically generated and maintained
reference models can help to get precise system specification
Motivation [1]
Volker Fortströer - Learning by Testing 3
Model generation from source code is possible but not practicable ◦ hardware systems, closed source components
An automated method is needed to generate a model of a (reactive) system
Motivation [2]
Volker Fortströer - Learning by Testing 4
Reactive systems Regular languages Finite-state automaton Learning an automaton Angluins Algorithm L* Optimizations to L*
Overview
Volker Fortströer - Learning by Testing 5
Systems which creates output on different input and may change internal states◦ i.e.: telecommunication systems
Preconditions:◦ The system responses have to be deterministic◦ The same input results in the same end state (output)◦ All input is accepted regardless of the system’s internal
state Conclusion: such a system can be seen as an
input/output automaton This automaton can be learned through algorithms
by creating a deterministic automaton
Reactive Systems
Volker Fortströer - Learning by Testing 6
A deterministic finite-state automaton (DFA) over Σ is a structure A = (Q, δ, q0, F).◦ Q: non-empty finite set of states◦ q0 є Q: initial state◦ F ⊆ Q: set of final states◦ δ: Q х Σ → Q is the transition function
Run of A: execution of A with input w◦ w: a1 … an є Σ*◦ Acceptance of w if A stops in a state q є F◦ The set of accepted words build a regular language
Deterministic Finite-State Automaton
Volker Fortströer - Learning by Testing 7
Let L be a language over the alphabet Σ. L ⊆ Σ* is called regular iff. either
◦ L is generated by a regular grammar or◦ L is accepted by a finite automaton or◦ L can be described trough a regular expression
Regular languages
Volker Fortströer - Learning by Testing 8
Algorithm L* invented by Angluin (87)◦ determining an initially unknown regular language
L◦ Two information sources are needed to learn the
regular language L: Membership Oracle (MO) checks if a sequence w is in
L Equivalence Oracle (EO) checks if an acceptor for L
is equivalent to the unknown L◦ Construct a equivalent minimal DFA which
accepts L
Learning a Finite Automata [1]
Volker Fortströer - Learning by Testing 9
Basic idea behind L*:◦ Systematically explore feasible strings using the
MO◦ Construct the transition table◦ maintain during exploration
A set S of state-access strings A set E of explored states A function T for mapping (S ∪ S ∙ A) ∙ E → {0,1}
T is kept in the observation table (OT) A is the input-alphabet
Learning a Finite Automata [2]
Volker Fortströer - Learning by Testing10
◦ Gradually build a DFA M (hypothesis)◦ If M seems to be stable make an equivalence
query If answer is correct then we have found an
equivalent DFA Otherwise we get a counterexample (which can be
used for refining DFA M)
Learning a Finite Automata [3]
Volker Fortströer - Learning by Testing
Assume a reactive system like a telecommunication system
One switch and some phones Phone actions (input):
◦ on-hook (↑)◦ off-hook (↓)◦ perform a call (→)
Switch actions (output):◦ initiated◦ cleared◦ hookswitch (signal received)
11
Application of L* [1]
Volker Fortströer - Learning by Testing
different (simple) scenarios:◦ S1: 1 physical device A
AI = {A↑, A↓}AO = {initiatedA, clearedA, [hookswitchA]}
◦ S2: 2 physical device A, BAI = {A↑, A↓, B↑, B↓}AO = {initiatedA,B, clearedA,B, [hookswitchA,B]}
depending on input through devices the switch produces output corresponding to its internal states
⇒ I/O-Automaton
12
Application of L* [2]
Volker Fortströer - Learning by Testing
An input / output automaton over Σ is a structure S = (Σ, AI, AO, →, s0).◦ Σ: non-empty finite set of states◦ AI: finite set of input actions◦ AO: finite set of output responses◦ →: transition relation → ⊆ Σ х AI х AO* х Σ ◦ s0 є Σ: initial state
An input / output automaton can be transformed in a deterministic finite-state automaton◦ Divide input words into single symbols◦ Create auxiliary-states for transitions
13
Input / Output Automaton
Volker Fortströer - Learning by Testing14
Represent a I/O-automaton as a DFA◦ Because a large input alphabet results in
inefficient runtime of L* Membership Oracle:
◦ This queries can be answered by the system itself◦ Problems:
Interface to the system is needed for invoking the input und capturing the response
Expensive in time because of timeouts
Adaptations to L* for learning a reactive system [1]
Volker Fortströer - Learning by Testing
Equivalence Oracle:◦ Learning a black-box system
⇒ one can never be sure that the whole system behavior was learned
◦ Using an approximation: Checking consistency within a fixed look ahead from
all states
15
Adaptations to L* for learning a reactive system [2]
Volker Fortströer - Learning by Testing16
Performing an oracle query takes relatively much time◦ L* has to wait for response of tested system
An obviously improvement is reducing the number of queries
Some queries can be answered with the result of other queries◦ i.e.: counterexample of EQ
Other queries can be filtered because of the properties of the learned language
Optimizing the runtime of L*
Volker Fortströer - Learning by Testing17
If every state is an accepting state the corresponding language is prefix-closed◦ All prefixes v of every word w in language L are
member of L Filter 1 (positive prefix):
∃σ2 ∈ A*. T(σ1;σ2) = 1 ⇒ MO(σ1) = true Filter 2 (negative prefix):
∃σ2 ∈ prefix(σ1). T(σ2) = 0 ⇒ MO(σ1) = false
Prefix-Closure
Volker Fortströer - Learning by Testing18
Every input produces always the same output
Filter 3 (Input Determinism):∃x ∈ AO, y ∈ A, σ2, σ3 ∈ A*. σ1 = σ2;x;σ3 ∧ T(σ2;y;σ3 ) = 1 ∧ x ≠ y ⇒ MO(σ1) = false
Filter 4 (Output Completion): ∃a ∈ AI, x ∈ AO. σ2;x ∈ prefix(σ1) ∧ T(σ2;a) = 1 ⇒ MO(σ1) = false
Input Determinism
Volker Fortströer - Learning by Testing19
Reactive systems exhibit often a high degree of parallelism
Sometimes different components of one type are interchangeable◦ i.e.: a device A behaves like device B◦ Independence:
actions of device A and B can be performed in different order
◦ Symmetry:an action performed by A can also be performed by B
Independence of Events [1]
Volker Fortströer - Learning by Testing
A membership query for σ1 is true if an equivalent σ2 is already in OT
Let Cpo,sym be the set of all equivalent sequences up to the given partial order and symmetry
Filter 5 (Partial Order): ∃σ2 ∈ Cpo,sym(σ1). T(σ2) = 1 ⇒ MO(σ1) = true
20
Independence of Events [2]
Volker Fortströer - Learning by Testing
The latter filter depends strongly on the concrete application domain,but there exist an pattern for realization:
1. An expert specifies an independence relation2. Abstraction by replacing concrete identifiers
with generic place holders3. σ is inspected if it contains independent
subparts4. All re-orderings are computed5. Generic place holders are replaced again by
concrete identifiers
21
Independence Pattern
Volker Fortströer - Learning by Testing
remember the two scenarios S1, S2 Number of Membership Queries:
relatively great effect other scenario with 80 states has a total
factor of 459.5 after applying the filters 22
Conclusion
States
no filter
1 & 2
Factor 3 & 4
Factor 5 Factor total Factor
S1 4 108 30 3.6 15 2.0 14 1.1 7.7S2 12 2431 593 4.1 218 2.7 97 2.2 25.1
Volker Fortströer - Learning by Testing
Thanks for your attention!
References:◦ Hardi Hungar, Oliver Niese, Bernhard Steffen: Domain-Specific
Optimization in Automata Learning. CAV 2003: 315-327◦ Therese Berg, Bengt Jonsson, Martin Leucker, and Mayank Saksena (SVV
2003), Mumbai (India): Insights to Angluin's learning. Electr. Notes Theor. Comput. Sci. 118: 3-18 (2005).
23