tao xie north carolina state university in collaboration with nikolai tillmann, peli de halleux,...
TRANSCRIPT
![Page 1: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/1.jpg)
Automated Developer Testing:Achievements and ChallengesTao Xie
North Carolina State University
In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU
ASE
![Page 2: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/2.jpg)
Why Automate Testing?
Software testing is important Software errors cost the U.S. economy about $59.5
billion each year (0.6% of the GDP) [NIST 02] Improving testing infrastructure could save 1/3 cost
[NIST 02] Software testing is costly
Account for even half the total cost of software development [Beizer 90]
Automated testing reduces manual testing effort Test execution: JUnit, NUnit, xUnit, etc. Test generation: Pex, AgitarOne, Parasoft Jtest, etc. Test-behavior checking: Pex, AgitarOne, Parasoft Jtest, etc.
![Page 3: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/3.jpg)
Automation in Developer Testing Developer testing
http://www.developertesting.com/ Kent Beck’s 2004 talk on “Future of
Developer Testing”http://www.itconversations.com/shows/detail301.html
This talk focuses on tool automation in developer testing (e.g., unit testing) Not system testing etc. conducted by testers
![Page 4: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/4.jpg)
Software Testing Setup
=?
Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
![Page 5: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/5.jpg)
Software Testing Problems
=?
Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
![Page 6: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/6.jpg)
Software Testing Problems
=?
Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Generating high-quality test inputs (e.g.,
achieving high code coverage)
Test Oracles Specifying high-quality test oracles (e.g.,
guarding against various faults)
![Page 7: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/7.jpg)
The Recipe of Unit Testing
var list = new List(); list.Add(item); var count = list.Count;
Assert.AreEqual(1, count);}
Three essential ingredients: Data Method Sequence Assertionsvoid Add() { int item = 3;
![Page 8: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/8.jpg)
The (problem with) Data
Which value matters? Bad choices cause incomplete test
suites. Hard-coded values get stale when
product code changes. Why pick a value if it doesn’t matter?
list.Add(3);
![Page 9: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/9.jpg)
Parameterized Unit Testing
void Add(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}
Parameterized Unit Test = Unit Test with Parameters
Separation of concerns Data is generated by a tool Developer can focus on functional
specification
[Tillmann&Schulte ESEC/FSE 05]
![Page 10: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/10.jpg)
Parameterized Unit Tests areAlgebraic Specifications
A Parameterized Unit Test can be read as a universally quantified, conditional axiom.void ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);} string name, string data: name ≠ null data ≠ null ⋀ ⇒ equals( ReadResource(name,WriteResource(name,data)), data)
![Page 11: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/11.jpg)
Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly
supported by various test frameworks .NET: Supported by .NET test frameworks
http://www.mbunit.com/ http://www.nunit.org/ …
Java: Supported by JUnit 4.X http://www.junit.org/
Generating test inputs for PUTs supported by tools .NET: Supported by Microsoft Research Pex
http://research.microsoft.com/Pex/ Java: Supported by Agitar AgitarOne
http://www.agitar.com/
![Page 12: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/12.jpg)
Test Generation
Human Expensive, incomplete, …
Brute Force Pairwise, predefined data, etc…
Random: Cheap, Fast “It passed a thousand tests” feeling
Dynamic Symbolic Execution: Pex, CUTE,EXE Automated white-box Not random – Constraint Solving
![Page 13: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/13.jpg)
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==1234567890
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
Observed constraints
a==nulla!=null &&!(a.Length>0)a!=null &&a.Length>0 &&a[0]!=1234567890
a!=null &&a.Length>0 &&a[0]==1234567890
Data
null
{}
{0}
{123…}a==null
a.Length>0
a[0]==123…T
TF
T
F
F
Execute&MonitorSolve
Choose next path
Done: There is no path left.
Negated condition
![Page 14: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/14.jpg)
Loops Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri AST 09]
Opportunities Regression testing [Taneja et al. ICSE 09-nier] Developer guidance (cooperative developer testing)
Challenges of DSE
![Page 15: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/15.jpg)
Loops Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri AST 09]
Applications Test network app at Army division@Fort Hood, Texas Test DB app of hand-held medical assistant device at
FDA
NCSU Industry Tech Transfer
![Page 16: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/16.jpg)
Pex on MSDN DevLabsIncubation Project for Visual Studio
Download counts (20 months)(Feb. 2008 - Oct. 2009 )
Academic: 17,366 Devlabs: 13,022 Total: 30,388
![Page 17: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/17.jpg)
Loops Fitnex [Xie et al. DSN 09]
Generic API functions e.g., RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences MSeqGen [Thummalapenta et al. ESEC/FSE 09]
Environments e.g., file systems, network, db, … Parameterized Mock Objects [Marri AST 09]
Applications Test network app at Army division@Fort Hood, Texas Test DB app of hand-held medical assistant device at FDA
NCSU Industry Tech Transfer
![Page 18: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/18.jpg)
There are decision procedures for individual path conditions, but… Number of potential paths grows
exponentially with number of branches Reachable code not known initially Without guidance, same loop might be
unfolded forever
Fitnex search strategy [Xie et al. DSN 09]
Explosion of Search Space
![Page 19: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/19.jpg)
public bool TestLoop(int x, int[] y) {if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
Path condition:!(x == 90) ↓New path condition:(x == 90) ↓New test input:TestLoop(90, {0})
Test input:TestLoop(0, {0})
DSE Example
![Page 20: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/20.jpg)
Path condition:(x == 90) && !(y[0] == 15) ↓New path condition:(x == 90) && (y[0] == 15) ↓New test input:TestLoop(90, {15})
Test input:TestLoop(90, {0})
public bool TestLoop(int x, int[] y) {if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
DSE Example
![Page 21: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/21.jpg)
public bool TestLoop(int x, int[] y) {if (x == 90) {
for (int i = 0; i < y.Length; i++)if (y[i] == 15) x++;
if (x == 110)return true;
} return false;}
Test input:TestLoop(90, {15})
Path condition:(x == 90) && (y[0] == 15) && !(x+1 == 110) ↓New path condition:(x == 90) && (y[0] == 15) && (x+1 == 110) ↓New test input:No solution!?
Challenge in DSE
![Page 22: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/22.jpg)
public bool TestLoop(int x, int[] y) {if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
Path condition:(x == 90) && (y[0] == 15) && (0 < y.Length) && !(1 < y.Length) && !(x+1 == 110) ↓New path condition:(x == 90) && (y[0] == 15) && (0 < y.Length) && (1 < y.Length)
Expand array size
Test input:TestLoop(90, {15})
A Closer Look
![Page 23: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/23.jpg)
We can have infinite paths! (both length and number)
Manual analysis need at least 20 loop iterations to cover the target branch
Exploring all paths up to 20 loop iterations is practically infeasible: 220 paths
Test input:TestLoop(90, {15})
public bool TestLoop(int x, int[] y) {if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
A Closer Look
![Page 24: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/24.jpg)
Fitnex: Fitness-Guided Exploration
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
Test input:TestLoop(90, {15, 15})
Our solution: Prefer to flip nodes on the most promising path Prefer to flip the most promising nodes on path Use fitness function as a proxy for promising
Key observations: with respect to the coverage target, not all paths are equally
promising for flipping nodes not all nodes are equally
promising to flip
![Page 25: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/25.jpg)
Fitness Function
FF computes fitness value (distance between the current state and the goal state)
Search tries to minimize fitness value
[Tracey et al. 98, Liu at al. 05, …]
![Page 26: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/26.jpg)
Fitness Function for (x == 110)
public bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++)
if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
Fitness function: |110 – x |
![Page 27: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/27.jpg)
Compute Fitness Values for Paths
(90, {0}) 20(90, {15}) 19(90, {15, 0}) 19(90, {15, 15}) 18(90, {15, 15, 0}) 18(90, {15, 15, 15}) 17(90, {15, 15, 15, 0}) 17(90, {15, 15, 15, 15}) 16(90, {15, 15, 15, 15, 0}) 16(90, {15, 15, 15, 15, 15}) 15…
FitnessValue
(x, y)
Give preference to flip a node in paths with better fitness values.We still need to address which node to flip on paths …
public bool TestLoop(int x, int[] y) {
if (x == 90) { for (int i = 0; i < y.Length; i+
+) if (y[i] == 15) x++;
if (x == 110) return true;
} return false;} Fitness function: |110
– x |
![Page 28: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/28.jpg)
FitnessValue
Compute Fitness Gains for Branches
public bool TestLoop(int x, int[] y) {
if (x == 90) { for (int i = 0; i < y.Length; i+
+) if (y[i] == 15) x++;
if (x == 110) return true;
} return false;}
(90, {0}) 20(90, {15}) flip b4 19(90, {15, 0}) flip b2 19(90, {15, 15}) flip b4 18(90, {15, 15, 0}) flip b2 18(90, {15, 15, 15}) flip b4 17(90, {15, 15, 15, 0}) flip b2 17(90, {15, 15, 15, 15}) flip b4 16(90, {15, 15, 15, 15, 0}) flip b2 16(90, {15, 15, 15, 15, 15}) flip b4 15…
(x, y)
Fitness function: |110 – x |Branch b1: i < y.LengthBranch b2: i >= y.Length
Branch b3: y[i] == 15Branch b4: y[i] != 15
• Flipping branch node of b4 (b3) gives us average 1 (-1) fitness gain (loss)
• Flipping branch node of b2 (b1) gives us average 0 (0) fitness gain (loss)
![Page 29: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/29.jpg)
Let p be an already explored path, and n a node on that path, with explored outgoing branch b.
After (successfully) flipping n, we get path p’ that goes to node n, and then continues with a different branch b’.
Define fitness gains as follows, where F(.) is the fitness value of a path. Set FGain(b) := F(p) – F(p’) Set FGain(b’) := F(p’) – F(p)
Compute the average fitness gain for each program branch over time
Compute Fitness Gains for Branches
![Page 30: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/30.jpg)
Implementation in Pex
Pex: Automated White-Box Test Generation tool for .NET, based on Dynamic Symbolic Execution
Pex maintains global search frontier All discovered branch nodes are added to frontier Frontier may choose next branch node to flip Fully explored branch nodes are removed from
frontier Pex has a default search frontier
Tries to create diversity across different coverage criteria, e.g. statement coverage, branch coverage, stack traces, etc.
Customizable: Other frontiers can be combined in a fair round-robin scheme
![Page 31: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/31.jpg)
Implementation in Pex
We implemented a new search frontier “Fitnex”:
Nodes to flip are prioritized by their composite fitness value:
F(pn) – FGain(bn),
where pn is path of node n bn is explored outgoing branch of n
Fitnex always picks node with lowest composite fitness value to flip.
To avoid local optimal or biases, the fitness-guided strategy is combined with Pex’s search strategies
![Page 32: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/32.jpg)
A collection of micro-benchmark programs routinely used by the Pex developers to evaluate Pex’s performance, extracted from real, complex C# programs
Ranging from string matching like
if (value.StartsWith("Hello") && value.EndsWith("World!") && value.Contains(" ")) { … }
to a small parser for a Pascal-like language where the target is to create a legal program.
Evaluation Subjects
![Page 33: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/33.jpg)
Pex with the Fitnex strategy Pex without the Fitnex strategy
Pex’s previous default strategy Random
a strategy where branch nodes to flip are chosen randomly in the already explored execution tree
Iterative Deepening a strategy where breadth-first search is
performed over the execution tree
Search Strategies Under Comparison
![Page 34: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/34.jpg)
#runs/iterations required to cover the target
Pex w/o Fitnex: avg. improvement of factor 1.9 over RandomPex w/ Fitnex: avg. improvement of factor 5.2 over Random
Evaluation Results
![Page 35: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/35.jpg)
Object Creation
Pex normally uses public methods to configure non-public object fields
Heuristics built-in to deal with common types
User can help if neededvoid (Foo foo) { if (foo.Value == 123) throw …[PexFactoryMethod]Foo Create(Bar bar) { return new Foo(bar);}
![Page 36: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/36.jpg)
QuickGraph Example
A graph example from QuickGraph library
interface IGraph{ /* Adds given vertex to the graph */ void AddVertex(IVertex v);
/* Creates a new vertex and adds it to the graph */ IVertex AddVertex();
/* Adds an edge to the graph. Both vertices should already exist in the graph */ IEdge AddEdge(IVertex v1, Ivertex v2); }
![Page 37: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/37.jpg)
Method Under Test
Desired object state for reaching targets 1 and 2: graph object should contain vertices and edges
Class SortAlgorithm{ IGraph graph; public SortAlgorithm(IGraph graph) { this.graph = graph; } public void Compute (IVertex s) { foreach(IVertex u in graph.Vertices) { //Target 1 }
foreach(IEdge e in graph.Edges) { //Target 2 } }}
method sequence
![Page 38: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/38.jpg)
Method Under Test
VertexAndEdgeProvider v0 = new VertexAndEdgeProvider();Boolean v1 = false;BidirectionalGraph v2 = new BidirectionalGraph((IVertexAndEdgeProvider)v0, v1);IVertex v3 = v2.AddVertex();IVertex v4 = v0.ProvideVertex();IEdge v15 = v2.AddEdge(v3, v4);
Achieved 31.82% (7 of 22) branch coverageReason for low coverage: Not able to generate graph with vertices and edges
Applying Randoop, a random testing approach that constructs test inputs by randomly selecting method calls
Example sequence generated by Randoop
v4 not in the graph, so edge cannot be added
to graph.
![Page 39: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/39.jpg)
New MSeqGen Approach
VertexAndEdgeProvider v0;bool bVal;IGraph ag = new AdjacencyGraph(v0, bVal);IVertex source = ag.AddVertex();IVertex target = ag.AddVertex();IVertex vertex3 = ag.AdVertex();IEdge edg1 = ag.AddEdge(source, target);IEdge edg2 = ag.AddEdge(target, vertex3);IEdge edg3 = ag.AddEdge(source, vertex3);
Use mined sequences to assist Randoop and PexBoth Randoop and Pex achieved 86.40% (19 of 22) branch coverage with assistance from MSeqGen
Mine sequences from existing code bases Reuse mined sequences for achieving desired object states
A Mined sequence from an existing codebase
Graph object includes both vertices and edges
![Page 40: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/40.jpg)
Challenges Addressed by MSeqGen
Existing codebases are often large and complete analysis is expensive Search and analyze only relevant portionsConcrete values in mined sequences may be different from desired values Replace concrete values with symbolic values
and use dynamic symbolic executionExtracted sequences individually may not be sufficient to achieve desired object states Combine extracted sequences to generate new
sequences
![Page 41: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/41.jpg)
MSeqGen: Code Searching
Problem: Existing code bases are often large and complete analysis is expensive Solution:
Use keyword search for identifying relevant method bodies using target classesAnalyze only those relevant method bodies
Target classes: System.Collections.Hashtable QuickGraph.Algorithms.TSAlgorithm
Keywords: Hashtable, TSAlgorithm
Shortnames of target classes are used as
keywords
![Page 42: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/42.jpg)
MSeqGen: Sequence Generalization
Problem: Concrete values in mined sequences are different from desired values to achieve target statesSolution: Generalize sequences by replacing concrete values with symbolic values
Class A { int f1 { set; get; } int f2 { set; get; } void CoverMe() { if (f1 != 10) return; if (f2 > 25) throw new Exception(“bug”); }}
Method Under Test
A obj = new A();obj.setF1(14);obj.setF2(-10);obj.CoverMe();
Mined Sequence for A
Sequence cannot help in exposing bug since desired values are f1=10 and f2>25
![Page 43: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/43.jpg)
MSeqGen: Sequence Generalization
Replace concrete values 14 and -10 with symbolic values X1 and X2
A obj = new A();obj.setF1(14);obj.setF2(-10);obj.CoverMe();
Mined Sequence for A int x1 = *, x2 = *;
A obj = new A();obj.setF1(x1);obj.setF2(x2);obj.CoverMe();
Generalized Sequence for A
Use DSE for generating desired values for X1 and X2DSE explores CoverMe method and generates desired values (X1 = 10 and X2 = 35)
![Page 44: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/44.jpg)
Improvement of State-of-the-Art Randoop
Without assistance from MSeqGen: achieved 32% branch coverage achieved 86% branch coverage
In evaluation, help Randoop achieve 8.7% (maximum 20%) higher branch coverage
Pex Without assistance from MSeqGen: achieved 45%
branch coverage achieved 86% branch coverage In evaluation, help Pex achieve 17.4% (maximum 22.5%)
higher branch coverage
![Page 45: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/45.jpg)
Test Oracles
Write assertions and Pex will try to break them
Without assertions, Pex can only find violations of runtime contracts causing NullReferenceException, IndexOutOfRangeException, etc.
Assertions leveraged in product and test code
Pex can leverage Code Contracts
![Page 46: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/46.jpg)
Summary:Automated Developer Testing
=?
Outputs
Expected
Outputs
Program
+
Test inputs
Test Oracles
Test Generation Test inputs for PUT generated by tools (e.g., Pex)
Fitnex: guided exploration of paths [DSN 09] MSeqGen: exploiting real-usage sequences
[ESEC/FSE 09]
Test Oracles Assertions in PUT specified by developers
Division of Labors
![Page 47: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/47.jpg)
Thank you
http://research.microsoft.com/pexhttp://pexase.codeplex.com/ https://sites.google.com/site/asergrp/
![Page 48: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/48.jpg)
Code Contracts
http://research.microsoft.com/en-us/projects/contracts/
Library to state preconditions, postconditions, invariants
Supported by two tools: Static Checker Rewriter: turns Code Contracts into runtime
checks Pex analyses the runtime checks
Contracts act as Test Oracle Pex may find counter examples for
contracts Missing Contracts may be suggested
![Page 49: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/49.jpg)
Example: ArrayList
Class invariant specification:
public class ArrayList { private Object[] _items; private int _size; ...
[ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >= this._size);
}
![Page 50: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/50.jpg)
ParameterizedModels
![Page 51: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/51.jpg)
Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.
Integration test: exercises large portions of a system.
Observation: Integration tests are often “sold” as unit tests
White-box test generation does not scale well to integration test scenarios.
Possible solution: Introduce abstraction layers, and mock components not under test
Unit Testing vs. Integration Testing
![Page 52: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/52.jpg)
AppendFormat(null, “{0} {1}!”, “Hello”, “World”); “Hello World!”
.Net Implementation:
public StringBuilder AppendFormat( IFormatProvider provider, char[] chars, params object[] args) {
if (chars == null || args == null) throw new ArgumentNullException(…); int pos = 0; int len = chars.Length; char ch = '\x0'; ICustomFormatter cf = null; if (provider != null) cf =
(ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter)); …
ExampleTesting with Interfaces
![Page 53: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/53.jpg)
Introduce a mock class which implements the interface.
Write assertions over expected inputs, provide concrete outputs
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); }}
Problems: Costly to write detailed behavior by example How many and which mock objects do we need to
write?
Stubs / Mock Objects
![Page 54: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/54.jpg)
Parameterized Mock Objects - 1 Introduce a mock class which implements the
interface. Let an oracle provide the behavior of the mock
methods.
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); return o; }}
Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated! 54
![Page 55: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/55.jpg)
Chosen values can be shaped by assumptions
public class MFormatProvider : IFormatProvider {
public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); PexAssume.IsTrue(o is ICustomFormatter);
return o; }}
(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)
Parameterized Mock Objects - 2
55
![Page 56: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/56.jpg)
Models from Choices
Choices to build parameterized models
class PFileSystem : IFileSystem { // cached choices PexChosenIndexedValue<string,string> files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}
![Page 57: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/57.jpg)
57
57
Subjects: QuickGraph Facebook
Research Questions: RQ1: Can our approach assist Randoop (random
testing tool) in achieving higher code coverages? RQ2: Can our approach assist Pex (DSE-based
testing tool) in achieving higher code coverages?
MSeqGen Evaluation
![Page 58: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/58.jpg)
58
RQ1: Assisting Randoop
![Page 59: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/59.jpg)
59
RQ2: Assisting Pex
Legend: #c: number of classes P: branch coverage achieved by Pex P + M: branch coverage achieved by Pex and MSeqGen
![Page 60: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/60.jpg)
Assumptions and Assertions
void PexAssume.IsTrue(bool c) { if (!c) throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException();}
Assumptions and assertions induce branches
Executions which cause assumption violations are ignored, not reported as errors or test cases
![Page 61: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/61.jpg)
How to test this code?(Actual code from .NET base class libraries)
Motivation: Unit Testing HellResourceReader
![Page 62: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/62.jpg)
Motivation: Unit Testing HellResourceReader
![Page 63: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/63.jpg)
ResourceReader PUT
[PexClass, TestClass][PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(typeof(BadImageFormatException))][PexAllowedException(typeof(IOException))][PexAllowedException(typeof(NotSupportedException))]public partial class ResourceReaderTest { [PexMethod] public unsafe void ReadEntries(byte[] data) { PexAssume.IsTrue(data != null); fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream(p, data.Length)) { var reader = new ResourceReader(stream); foreach (var entry in reader) { /* reading entries */ } } }}
![Page 64: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/64.jpg)
Creating Complex Objects Exploration of constructor/mutator
method sequences
Testing with class invariants
![Page 65: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/65.jpg)
Write class invariant as boolean-valued parameterless method Refers to private fields Must be placed in implementation code
Exploration of valid states by setting public/private fields May include states that are not reachable
Object Creation:Class Invariants
![Page 66: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/66.jpg)
Example: ArrayList
Class invariant specification:
public class ArrayList { private Object[] _items; private int _size; ...
[ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >= this._size);
}
![Page 67: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/67.jpg)
Example: ArrayList
PUT:
[PexMethod]public void ArrayListTest(ArrayList al, object o){ int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}
![Page 68: Tao Xie North Carolina State University In collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU](https://reader038.vdocuments.mx/reader038/viewer/2022110319/56649c755503460f94928f75/html5/thumbnails/68.jpg)
Example: ArrayList
Generated Test:
[TestMethod] public void Add01() { object[] os = new object[0]; // create raw instance ArrayList arrayList = PexInvariant.CreateInstance<ArrayList>();
// set private field via reflection PexInvariant.SetField<object[]>(arrayList, "_items", os); PexInvariant.SetField<int>(arrayList, "_size", 0);
// invoke invariant method via reflection PexInvariant.CheckInvariant(arrayList);
// call to PUT ArrayListTest(arrayList, null); }