a theoretical framework for understanding mutation-based testing methods
TRANSCRIPT
![Page 1: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/1.jpg)
A Theoretical Framework for Understanding Mutation-Based Testing Methods
Donghwan Shin and Doo-Hwan Bae
KAIST, South Korea
@ ICST 2016
![Page 2: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/2.jpg)
© Donghwan Shin, 2016
Mutation-based testing has been widely studied for addressing various testing problems
Test set selection Fault localization Program repair
2
𝑝𝑜 𝑚𝑧
𝑚𝑥
𝑚𝑦
Artificially mutated program𝒑𝒐: an original program𝒎: a mutant generated from 𝒑𝒐
Artificial fault Partial fix
Systematically generate 𝒎 from 𝒑𝒐
and use the differences between them
difference difference
difference
![Page 3: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/3.jpg)
© Donghwan Shin, 2016
There is a solid theoretical framework for general testing process – “testing system”
Such theoretical framework facilitates a clear understanding of the essence of complex problems
3
𝑷 attempts to implement 𝑺
𝑻 is designed to consider 𝑺 and 𝑷
𝑶 determines the correctness of 𝑷 for 𝑻
⋯
𝑃𝑆
𝑇𝑂
𝑷 is a set of programs 𝑺 is a set of specifications 𝑻 is a set of tests 𝑶 is a set of oracles
Fundamental testing factors and their relationships[1][2]
[1] J. S. Gourlay, “A mathematical framework for the investigation of testing,” Software Engineering, IEEE Transactions on, no. 6, pp. 686-709, 1983.[2] M. Staats, M. W. Whalen, and M. P. E. Heimdahl, “Programs, tests, and oracles: the foundations of testing revisited,” in Proceedings of ICSE 2011, pp. 391-400.
![Page 4: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/4.jpg)
© Donghwan Shin, 2016
Surprisingly little attention has been paid to the theoretical framework of mutation-based testing
4
Even the testing system is not adequate to mutation-based testing because it focuses on the correctness.
𝑷 attempts to implement 𝑺
𝑻 is designed to consider 𝑺 and 𝑷
𝑶 determines the correctness of 𝑷 for 𝑻
⋯
𝑃𝑆
𝑇𝑂
𝑺 is a set of specifications 𝑷 is a set of programs 𝑻 is a set of tests 𝑶 is a set of oracles
How to formally describethe behavioral differences between programs and its mutants in testing?
𝑀
![Page 5: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/5.jpg)
© Donghwan Shin, 2016
Key idea: a new testing factor for difference
5
𝑜 𝑡, 𝑝 = 1 𝑡𝑟𝑢𝑒 ,0 𝑓𝑎𝑙𝑠𝑒 ,
if 𝒑 is correct for 𝒕
otherwise
In the testing system, an oracle 𝒐 implies the correctness of a program 𝒑 for a test 𝒕 as follows:
Let us define a new testing factor 𝑿to imply the difference between two programs for a test
𝑋(𝑡, 𝑝𝑥, 𝑝𝑦)
![Page 6: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/6.jpg)
© Donghwan Shin, 2016
A theoretical framework for the difference in mutation-based testing
6
In this talk:• Define and extend a new testing factor to formalize the
“difference-based testing framework”
What you can do using this framework:• Formally describe the behavioral differences of programs
given a set of tests.
• Quantitatively represent and analyze the behavioral differences in a multi-dimensional space.
• Guide to understand mutation-based testing methods.
![Page 7: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/7.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
7
![Page 8: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/8.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
8
![Page 9: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/9.jpg)
© Donghwan Shin, 2016
Test differentiator: a new testing factor (1/2)
9
Def. 1: A test differentiator 𝑑 is a function such that
𝑑 𝑡, 𝑝𝑥, 𝑝𝑦 = 1 𝑡𝑟𝑢𝑒 ,0 𝑓𝑎𝑙𝑠𝑒 ,
if the behaviors of 𝒑𝒙 and 𝒑𝒚 are different for 𝒕
otherwise
𝑑 𝑡, 𝑝𝑜, 𝑚 = 1 “𝑡 detects the difference between 𝒑𝒐 and 𝒎”
Example
“The test kills the mutant”
![Page 10: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/10.jpg)
© Donghwan Shin, 2016
Test differentiator: a new testing factor (2/2)
It clarifies tricky concepts in mutation-based testing.
10
A test 𝑡 detects a fault in 𝑝𝑜 𝑑 𝑡, 𝑝𝑠, 𝑝𝑜 = 1
A test 𝑡 kills a mutant 𝑚 𝑑 𝑡, 𝑝𝑜, 𝑚 = 1
𝑝𝑠: the projection of the specification (i.e., true requirements of 𝑝𝑜)
It shows how 𝑑 is important in mutation-based testing.
𝑑 observes internal values
𝑑 observes only output
Weak mutation Strong mutation
![Page 11: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/11.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
11
![Page 12: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/12.jpg)
© Donghwan Shin, 2016
𝐝-vector: extended 𝑑 as a vector form
12
Def. 2: A 𝑑-vector 𝒅 is an 𝒏-dimensional vector, such that
𝐝 𝐭, 𝑝𝑥, 𝑝𝑦 = 𝑑 𝑡1, 𝑝𝑥, 𝑝𝑦 , … , 𝑑(𝑡𝑛, 𝑝𝑥, 𝑝𝑦)
for a collection of tests 𝐭 = 𝑡1, 𝑡2, ⋯ , 𝑡𝑛 ∈ 𝑻𝒏.
Test Behavior 𝐝-vector = behavioral difference
𝐭 𝑝𝑠 𝑝𝑜 𝑚 𝐝 𝐭, 𝑝𝑠, 𝑝𝑜 𝐝 𝐭, 𝑝𝑠, 𝑚 𝐝 𝐭, 𝑝𝑜, 𝑚
𝑡1 A A A
0,1,1,1 0,1,1,0 0,0,1,1𝑡2 B A A
𝑡3 C A D
𝑡4 D A D
𝑑-vector concisely describes the behavioral differences of programs for a set of tests
Example
![Page 13: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/13.jpg)
© Donghwan Shin, 2016
𝐝-vector = position in a multi-dimensional space
13
x
z
y
p(1,1,2)
A vector represents a point in a multi-dimensional space.
The position of a point is relative to the origin.
𝐝 𝐭, 𝑝𝑥, 𝑝𝑦= 𝑑1, … , 𝑑𝑛
We can think of a d-vector as the representation of a position in a space.
![Page 14: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/14.jpg)
© Donghwan Shin, 2016
Position and its space from a 𝐝-vector
14
Def. 3: The position of a program 𝒑𝒙 relative to another program 𝒑𝒓 in a multi-dimensional space corresponding to a set of tests 𝐭 is
𝐝 𝐭, 𝑝𝑟 , 𝑝𝑥 = 𝐝𝑝𝑟𝐭 𝑝𝑥 .
The space is induced by 𝐭, 𝑝𝑟 , 𝑑 .
𝐭 = 𝒕𝟏, ⋯ , 𝒕𝒏 corresponds to the 𝒏-dimensions.
𝑝𝑟 corresponds to the origin of the space.
𝑑 corresponds to the notion of difference in the space.
A position of 𝒑𝒙 relative to the origin 𝒑𝒓 implies the behavioral difference between 𝒑𝒙 and 𝒑𝒓 for a set of tests 𝐭.
![Page 15: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/15.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
15
![Page 16: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/16.jpg)
© Donghwan Shin, 2016
Position deviance relation (1/3)
Add more tests increasing dimensions make positions deviant from its origin
16
𝐭 = {⋯ , 𝑡𝑖 , 𝑡𝑖+1, ⋯ }
𝐩𝐱 = ⋯ , 0,0,⋯
𝐩𝐲 = ⋯ , 0,1,⋯
𝐭 = {⋯ , 𝑡𝑖 , ⋯ }
𝐩𝐱 = ⋯ , 0,⋯
𝐩𝐲 is deviant from 𝐩𝐱 by 𝑡𝑖+1
Example
![Page 17: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/17.jpg)
© Donghwan Shin, 2016
Position deviance relation (2/3)
Forms a partial order based on transitivity.
17
𝐩𝟏 = ⋯ , 0,0,⋯
𝐩𝟐 = ⋯ , 0,1,⋯
𝐩𝟑 = ⋯ , 1,1,⋯
𝐩𝟒 = ⋯ , 1,0,⋯
𝐩𝟑 is more deviant than 𝐩𝟏.
𝐩𝟑 and 𝐩𝟒 are incomparable.
![Page 18: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/18.jpg)
© Donghwan Shin, 2016
Position deviance relation (3/3)
If 𝑝𝑟 = 𝑝𝑜, the deviance relation on positions impliesthe dynamic subsumption* on mutants at the positions.
18
* Ammann, Paul, Marcio E. Delamaro, and Jeff Offutt. ”Establishing theoretical minimal sets of mutants.” ICST 2014.
Dynamic subsumptionIf 𝑚𝑥 is killed, then 𝑚𝑦 must be killed.
𝐩𝐱 = ⋯ , 0,0,⋯ = {𝑚𝑥}
𝐩𝐲 = ⋯ , 0,1,⋯ = {𝑚𝑦}
𝐩𝐨 = 0,⋯ , 0 = {𝑝𝑜}
![Page 19: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/19.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
19
![Page 20: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/20.jpg)
© Donghwan Shin, 2016
Position Deviance Lattice (PDL) (1/4)
20
It shows the positions with their deviance relation.• Each dimension (=test) has only two positions, 0 or 1.
• The number of positions in a 𝑛-dimensional space = 2𝑛.
𝐩0 = 0,0,0
𝐩2 = 0,1,0
𝐩5 = 1,0,1
𝐩7 = 1,1,1
𝐩1 = 1,0,0 𝐩3 = 0,0,1
𝐩4 = 1,1,0 𝐩6 = 0,1,1
𝑡1 𝑡2𝑡3
𝑡1 𝑡2 𝑡3
Dimensions
3-dimensional PDL
Example
![Page 21: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/21.jpg)
© Donghwan Shin, 2016
Position Deviance Lattice (PDL) (2/4)
21
It growths as more tests added.
𝐩0
𝐩2
𝐩5
𝐩7
𝐩1 𝐩3
𝐩4 𝐩6
𝑡1 𝑡2𝑡3
𝑡1 𝑡2 𝑡3
Dimensions
Example
𝐩𝟎, 𝐩𝟑
𝐩𝟐, 𝐩𝟔𝐩𝟏, 𝐩𝟓
𝐩𝟒, 𝐩𝟕
𝑡1 𝑡2
𝐩𝟎, 𝐩𝟐, 𝐩𝟑, 𝐩𝟔
𝐩𝟏, 𝐩𝟒, 𝐩𝟓, 𝐩𝟕
𝑡1
𝐭 = {𝑡1, 𝑡2} 𝐭 = {𝑡1, 𝑡2, 𝑡3}
𝐭 = {𝑡1}
![Page 22: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/22.jpg)
© Donghwan Shin, 2016
Outline
Test differentiator
• A new testing factor for the notion of difference
d-vector• Extended differentiator for a set of tests
Position• Redefined d-vector in a multi-dimensional space
Position deviance relation
• Formal relation between positions
Position Deviance Lattice• Graphical model for positions and its deviance relation
Applications
22
![Page 23: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/23.jpg)
© Donghwan Shin, 2016
Minimal set of mutants in PDL (1/2)
In PDL where 𝒑𝒓 = 𝒑𝒐, mutants in “the least deviant” positions form the minimal set of mutants.
23
𝐩0 = {𝑝𝑜}
𝐩2 = {}
𝐩5 = {𝑚3}
𝐩7 = {𝑚4}
𝐩1 = {𝑚1} 𝐩3 = {}
𝐩4 = {} 𝐩6 = {𝑚2}
𝑡1 𝑡2𝑡3
![Page 24: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/24.jpg)
© Donghwan Shin, 2016
Minimal set of mutants in PDL (2/2)
What is the maximum bound of the minimal set of mutants for 𝑛 tests?• Key: if two mutants are at two comparable positions, then
one must be dynamically subsumed by another.
• Sperner's theorem says that the maximum number of incomparable nodes in an 𝑛-dim lattice is given as follows:
24
max(|𝑀𝑚𝑖𝑛𝑖𝑚𝑎𝑙|) =𝑛
𝑛/2
(E.g.) If we have 10 tests, # of minimal mutants ≤ 𝟏𝟎𝟓
= 252
![Page 25: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/25.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
25
𝒅(𝒕𝒋, 𝒑𝒐,𝒎𝒊) 𝒎𝟏 𝒎𝟐 𝒎𝟑 𝒎𝟒
𝒕𝟏 1 1 0 0
𝒕𝟐 0 0 1 1
𝑝𝑜, 𝑚1, 𝑚2, 𝑚3, 𝑚4𝑇𝑆 = {}
![Page 26: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/26.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
26
𝑝𝑜, 𝑚3, 𝑚4
𝑚1, 𝑚2
𝑡1
𝑇𝑆 = {𝑡1}
𝒅(𝒕𝒋, 𝒑𝒐,𝒎𝒊) 𝒎𝟏 𝒎𝟐 𝒎𝟑 𝒎𝟒
𝒕𝟏 1 1 0 0
𝒕𝟐 0 0 1 1
![Page 27: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/27.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
27
𝑝𝑜
𝑚3, 𝑚4𝑚1, 𝑚2
𝑡1 𝑡2
(empty)
𝑇𝑆 = {𝑡1, 𝑡2}
𝒅(𝒕𝒋, 𝒑𝒐,𝒎𝒊) 𝒎𝟏 𝒎𝟐 𝒎𝟑 𝒎𝟒
𝒕𝟏 1 1 0 0
𝒕𝟐 0 0 1 1
![Page 28: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/28.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
28
𝑝𝑜
𝑚3, 𝑚4𝑚1, 𝑚2
𝑡1 𝑡2
(empty)
𝑇𝑆 = {𝑡1, 𝑡2}
Traditional mutation adequacy criterion• A test suite that distinguishes the positions of mutants from
the position of 𝒑𝒐will likely detect real faults.
![Page 29: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/29.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
29
𝑝𝑜
𝑚3, 𝑚4𝑚1, 𝑚2
𝑡1 𝑡2
(empty)
𝑇𝑆 = {𝑡1, 𝑡2}
Claim for the existing mutation adequacy criterion• In the “killed area”, there are still several mutants which are
not distinguished in terms of their positions.
![Page 30: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/30.jpg)
© Donghwan Shin, 2016
Not-killed area
Killed area
Understanding and extending the mutation adequacy criterion using PDL
30
𝑝𝑜
𝑚4
𝑚1
(empty)
𝑚2 (empty)
(empty) 𝑚3
𝑡1 𝑡2𝑡3
Distinguishing more mutants increases fault detection*• A test suite that distinguishes the positions of mutants from
each other will likely detect real faults.
* Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. “Diversity-aware mutation adequacy criterion for improving fault detection capability.” Mutation 2016
![Page 31: A Theoretical Framework for Understanding Mutation-Based Testing Methods](https://reader030.vdocuments.mx/reader030/viewer/2022020314/588020b51a28abbc128b6c43/html5/thumbnails/31.jpg)
© Donghwan Shin, 2016
Conclusion
Correctness-based think difference-based think• Incorrect programs seem useless, while different programs
seems meaningful.
PDL may guide you to consider difference-based think.
31
𝑡1 𝑡2 𝑡3
Dimensions