automated test data generation maili markvardt. outline introduction test data generation problem...
TRANSCRIPT
![Page 1: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/1.jpg)
Automated Test Data Generation
Maili Markvardt
![Page 2: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/2.jpg)
Outline
Introduction Test data generation problem Black-box approach White-box approach
![Page 3: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/3.jpg)
Introduction
Improtance of testing growing since mission-criticality of the software in our everyday life
Software errors are more costly than ever Testing can be automated
Test execution automation Test generation automation Test data generation automation
![Page 4: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/4.jpg)
Problem: example
User inputs three sides of a triangle (a, b, c). Which type is the triangle?
Requirements: IF a<=0 || b<=0 || c<=0 -> input incorrect IF p*(p-a)*(p-b)*(p-c) < 0 -> sides not forming a
triangle IF a==b || a==c || b ==c -> isoceles Kui a==b & b==c -> eqilateral Other -> scalene
What strategy? -> what data?
![Page 5: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/5.jpg)
Input validation automation
The Concept “side of a triangle” equivalence partitions and boundary values Normal: ]0; ∞[ Erroneous: ]- ∞; 0[, missing values Border values: {0}
For testing the Input validation functionality, pick a random value from each equivalence partition for each side: P(-1, 2, 3), P(1, -2, 3), P(1, 2, -3)
Same with boundary values P(0, 1, 2), P(1, 0, 2), P(1, 2, 0)
Input validation with “normal” values P(1, 2, 3)
![Page 6: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/6.jpg)
What about other requirements?
If input values are dependent and that affects output, random values can not be used! – we may not be able to find needed values with random generation if p*(p-a)*(p-b)*(p-c) < 0 -> sides don’t form a
triangle We must use specification-based (Black-box)
or program-based (White-box) test data generation
![Page 7: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/7.jpg)
Black-Box approach
Generating test data from formal specifications (ie. Z-notation)
Classification Tree Method (CTM) ...
![Page 8: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/8.jpg)
Classification Tree Method4
Based on equivalence partitions method: input and output properties are divided into equivalence partitions
Equivalence partitions are combined into test cases
The goal: minimal but sufficient amount of test cases
4Dai, Z. R., Deussen, P. H. Automatic Test Data Generation for TTCN-3 Using Classification Tree Method (2005)
![Page 9: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/9.jpg)
CTM
Equivalence partitions form a tree structure Input dependencies are not resolved
![Page 10: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/10.jpg)
White-Box approach
White-box test data generation – based on program structure
Test data generation problem: For program P and path u, find input x S, so that P(x) traverses path u, where S is the set of all input values
Remember: white-box approach is based on program formalisation (graph, FSM, …)
![Page 11: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/11.jpg)
Test data generator structure2
2Edvardsson, J. Contributions to Program- and Specification-based test data generation. (2002). www.ida.liu.se/~joned/papers/joned_lic.pdf
![Page 12: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/12.jpg)
Possible strategies (adequacy criteria)
Statement coverage Branch coverage Condition coverage Multiple-condition coverage Path coverage …
![Page 13: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/13.jpg)
Numerous methods for Constraint generator & Constraint solver Symbolic Execution Actual Execution Symbolic/Actual Execution hybrid Simulated Annealing Iterative Relaxation Technique Chaining Approach Genetic Algorithms MEA-Graph Planning ...
![Page 14: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/14.jpg)
Symbolic Execution2
Popular static method for finding path constraints Path constraints are rewritten using input variables
Not suitable for programs using pointers and arrays Not suitable for programs using precompiled units
read(a,b)
c=a+b;
d=a-b;
e=c*d;
if (e>5) {...}
a*a – b*b > 5=>
![Page 15: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/15.jpg)
Actual Execution2
Program is executed several times On every execution: check, whether or not
the desired path is executed If desired path is not executed, program is re-
executed with slightly modified input values Program is re-executed until desired path is
traversed or user-defined limit (time, execution count) is exceeded
Solves some problems of symbolic execution since values of variables are available
![Page 16: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/16.jpg)
Actual Execution
For each path condition bi objective function is found:
Fi(x) {<|<=|=} 0
If Fi(x) {<|<=|=} 0, then current path is executed
F(x)= Σ Fi(x), if branch consists of several conditions
How to minimize objective function so that Fi (x)=0
In other words, what input values are needed to execute desired path?
![Page 17: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/17.jpg)
Simulated Annealing
Simulated Annealing – generic probabilistic meta-algorithm for finding good approximation to the global optimum for a given function in a large search space
Analogy from metallurgy: Process of annealing is used for reducing defects in material Metal is heated: atoms start to move Metal is cooled down slowly: greater
probability that atoms find a “suitable” place
![Page 18: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/18.jpg)
Simulated Annealing
Goal: minimize the objective function -> desired path is executed
Find a “random” solution for objective function Compare the solution with current solution of
objective function Decide, whether or not the “random” solution
is better than current solution
![Page 19: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/19.jpg)
Simulated Annealing
If “random” solution gives a better value (closer to 0) for objective function, the “random” value is always chosen (probability is 1)
If “random” solution is not better than current solution – “sometimes” it is chosen – depends of the “temperature”
The value of “temperature” is decreased In the beginning high “temperature” -> almost every
solution is chosen When temperature is lowered, the probability of
choosing worse solution is lowered until it is 0
![Page 20: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/20.jpg)
Simulated Annealing: properties
Choosing worse solutions in the beginning lowers the probability of getting stuck in a local optimum (drawback of Gradient Descent/Hill Climbing/Greedy algorithms)
It is possible to show, that probability of finding global optimum is almost 1
Little use in practice, since finding the global optimum with sufficient significance by annealing takes more time than full search of the whole search space
![Page 21: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/21.jpg)
Simulated Annealing
Parameters for successful simulated annealing: Art rather than science How to find a “random” solution – how to
minimize the count of iterations finding the optimum?
How to determine, whether or not the “worse” solution is picked?
Annealing schedule – from what “temperature” to start and how the “temperature” is lowered?
![Page 22: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/22.jpg)
Genetic algorithms
Imitates the process of natural selection Evaluation Choice Recombination and mutation
Start with random set of solutions - population Solutions are evaluated for their fitness –
ability to generate good offspring Chosen (good) solutions are recombined and
mutated to generate a new generation of solutions
![Page 23: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/23.jpg)
GA for test data generation5
Algorithm is driven by control dependency graph of the program Graph nodes = program statements Graph edges = control dependencies between
program statements Goal: find data for executing certain node
(program sentence) Node X is post-dominated by node Y, if every
directed path from X to the end of the program includes node Y
5Pargas, R., Harrold, J.M., Peck, R.R. Test-Data generation Using genetic Algorithms (1999)
![Page 24: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/24.jpg)
GA for test data generation
Node Y is control dependent of Y, only if Exists a directed path from Y to X and all
nodes on this path (except X,Y) are post-dominated by Y and
X is not post-dominated by Y Control dependency predicate path (CDPP)–
predicates that must be satisfied on acyclic path from initial node to some other node X
![Page 25: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/25.jpg)
GA for TDG: algorithm
Solution is set of test data Start with random set Evaluate fitness of data
Execute the program with data, mark predicates on executed path
Compare the found set of predicates with CDPP to desired node
The more the found set of data allowed to execute CDPP, the better the data is
![Page 26: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/26.jpg)
GA for TDG
Best solutions are chosen, recombined and mutated to generate a new generation of solutions
Non-typical application of GA – several possible solutions, depending on the test goal ie. find data for executing nodes A, C – more
than one test may be needed if A and C are exclusive!
![Page 27: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/27.jpg)
GA: example
int i, j, k;
1: read i, j, k;
2: if (i<j) {
3: if (j<k) {
4: i=k;
} else {
5: k=i;
}
}
6: print i, j, k;5 is test goal,
CDGpath {ET, 2T, 3F}
![Page 28: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/28.jpg)
GA: example
Random population
Fitness f{2, 2, 0, 0} Probability of choosing pi = fi/Σfj ({0.5,0.5,0,0}) One solution can be chosen more than once
TestCase Input(i,j,k) Path
TC1 1, 6, 9 1,2,3,4,6
TC2 0, 1, 4 1,2,3,4,6
TC3 5, 0, 1 1, 2, 6
TC4 2, 2, 3 1, 2, 6
![Page 29: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/29.jpg)
GA: Näide
New population {(1, 6, 9), (0, 1, 4), (0, 1, 4), (0, 1, 4)}
Recombination (one-point crossover): n first values form one parent and others from the other parent N=2: {(1, 6, 4), (0, 1, 9)}
Mutation: Value in random position is replaced with a random number (0, 6, 4), (5, 1, 4)
![Page 30: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/30.jpg)
Summary
Numerous methods Black-Box White-Box
Choice of methods depends on knowledge and preference of tester, Technology of SUT
![Page 31: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/31.jpg)
Viited
1Edvardsson, J. A survey on Automatic Test Data Generation. (1999). [WWW] www.ida.liu.se/~joned/papers/class_atdg.pdf
2Edvardsson, J. Contributions to Program- and Specification-based test data generation. (2002). [WWW] www.ida.liu.se/~joned/papers/joned_lic.pdf
3Gupta, N., M, Mathur, A., Soffa, M.L. Automated Test Data Generation Using an Iterative Relaxation Method (1999)
![Page 32: Automated Test Data Generation Maili Markvardt. Outline Introduction Test data generation problem Black-box approach White-box approach](https://reader035.vdocuments.mx/reader035/viewer/2022081507/5518c28e550346991f8b563c/html5/thumbnails/32.jpg)
Viited
4Dai, Z. R., Deussen, P. H. Automatic Test Data Generation for TTCN-3 Using Classification Tree Method (2005)
5Pargas, R., Harrold, J.M., Peck, R.R. Test-Data generation Using genetic Algorithms (1999)