next generation developer testing: parameterized testing

Next Generation Developer Testing: Parameterized Testing

Tao Xie Department of Computer Science

University of Illinois at Urbana-Champaign Email: [email protected]

http://taoxie.cs.illinois.edu/

In Collaboration with Microsoft Research

mailto:[email protected]


2

Introduction

Developer (aka unit) testing – a widely adopted practice forensuring high quality software

A conventional unit test (CUT): small program with test inputs and test assertions

void AddTest() {

HashSet set = new HashSet();

set.Add(7);

set.Add(3);

Assert.IsTrue(set.Count == 2);

}

Test Scenario

Test Assertions

Test Data

3

Parameterized Unit Tests (PUTs)

Recent advances in unit testing introduced PUTs

void AddSpec(int x, int y)

{

HashSet set = new HashSet();

set.Add(x);

set.Add(y);

Assert.AreEqual(x == y, set.Count == 1);

Assert.AreEqual(x != y, set.Count == 2);

}

PUTs separate two concerns:• Specification of externally visible behavior (assertions)

• Selection of internally relevant test inputs (coverage)

4

Parameterized Unit Tests (PUTs)

More beneficial than CUTs

• Help describe behaviors for all test arguments

Address two main issues with CUTs

• Missing test data required for exercising importantbehaviors

Low fault-detection capability

• Including test data that exercises the same behaviour

Redundant unit tests

5

An Example using IntStack

public void CUT1() {int elem = 1;IntStack stk = new IntStack();stk.Push(elem);Assert.AreEqual(1, stk.Count());

}

Three CUTs


}

public void CUT3() {int elem1 = 1, elem2 = 30;IntStack stk = new IntStack();stk.Push(elem1);stk.Push(elem2);Assert.AreEqual(2, stk.Count());

}

CUT1 and CUT2 exercise push with different test data

CUT3 exercises push when stack is not empty

Two main issues with CUTs:

Fault-detection capability issue: undetected defect where things go wrong when passing a negative value to push

Redundant test issue: CUT2 is redundant with respect to CUT1

6

An Example using IntStack


}

Three CUTs


}


}

No need to describe test data

• Generated automatically

Single PUT replaces multiple CUTs

• With reduced size of test code

public void PUT(int[] elem) {Assume.IsTrue(elem != null);IntStack stk = new IntStack();for(int i in elem)

stk.push(elem);Assert.AreEqual(elem.Length, stk.count());

}

An equivalent PUT

Parameterized Unit Tests areAlgebraic Specifications

• A Parameterized Unit Test can be read as a universally quantified, conditional axiom.

void ReadWrite(Storage s, string name, string data) {Assume.IsTrue(s!=null && name!=null && data!=null);s.WriteResource(name, data);var readData = s.ReadResource(name);Assert.AreEqual(data, readData);

}

Storage s, string name, string data:s ≠ null ⋀ name ≠ null ⋀ data ≠ null ⇒equals(ReadResource(WriteResource(s,name,data).state, name).retval,data)

Parameterized Unit Testingis going mainstream

Parameterized Unit Tests (PUTs) commonly supported by various test frameworks

• .NET: Supported by .NET test frameworks

– http://www.mbunit.com/

– http://www.nunit.org/ …

• Java: Supported by JUnit 4.X

– http://www.junit.org/

Generating test inputs for PUTs supported by tools

• .NET: Supported by Microsoft Visual Studio 2015 IntelliTest

– Formerly Microsoft Research Pex: http://research.microsoft.com/pex/

• Java: Supported by Agitar AgitarOne

– http://www.agitar.com/

http://research.microsoft.com/pex/

http://www.agitar.com/

Parameterized Tests in JUnit

9https://github.com/junit-team/junit/wiki/Parameterized-tests

https://github.com/junit-team/junit/wiki/Parameterized-tests

JUnit Theories

10https://github.com/junit-team/junit/wiki/Theories

https://github.com/junit-team/junit/wiki/Theories

Assumptions and Assertions

void PexAssume.IsTrue(bool c) {if (!c)

throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) {

if (!c)throw new AssertionViolationException();

}

• Assumptions and assertions induce branches

• Executions which cause assumption violations are ignored, not reported as errors or test cases

Test Data Generation

• Human– Expensive, incomplete, …

• Brute Force– Pairwise, predefined data, etc…

• Semi - Random– Cheap, Fast– “It passed a thousand tests” feeling

• Dynamic Symbolic Execution: IntelliTest/Pex, SAGE, CUTE, …

– Automated white-box– Not random – Constraint Solving

13

void CoverMe(int[] a){

if (a == null) return;if (a.Length > 0)if (a[0] == 1234567890)

throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

T

Constraints to solve

a!=null

a!=null &&

a.Length>0

a!=null &&

a.Length>0 &&

a[0]==123456890

Input

null

{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==null

a!=null &&

!(a.Length>0)

a==null &&

a.Length>0 &&

a[0]!=1234567890

a==null &&

a.Length>0 &&

a[0]==1234567890

Generates test data systematically

Done: There is no path left.

Background: DSE

Pex4Funhttp://pex4fun.com/

Nikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith Bishop. Teaching and Learning Programming and Software Engineering via Interactive Gaming. In ICSE 2013, SEE.http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

1,703,247 clicked 'Ask Pex!'

http://pex4fun.com/

http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

Pex4Fun

• Click http://pex4fun.com/default.aspx?language=CSharp&sa

mple=_Template

• Copy and modify the following code snippet in the code

editing box (or simply click here)using System;

using Microsoft.Pex.Framework;

using Microsoft.Pex.Framework.Settings;

[PexClass]

public class Program

{

[PexMethod]//[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)]

public static string testMethod(int x, int y)

{

PexAssume.IsTrue(y >= 0);//replace here with your assumption

//... enter your code under test here

//if (x == 10000) throw new Exception();

PexAssert.IsTrue(y >= 0);//replace here with your assertion

return PexSymbolicValue.GetPathConditionString();

}

}

https://sites.google.com/site/teachpex/Home/pex-usage-tips

http://pex4fun.com/default.aspx?language=CSharp&sample=_Template

http://pex4fun.com/default.aspx?language=CSharp&code=agKVkcFqg0AQhu/6EnNUKGt6FgshJKWHQMDQS8nB6CQu1d2wO4tKybvkWfJknTUJacmlHcRxdub/92N0Vqo95IMlbNMQLhG48XQpS6Ot3pFYYS8Wpmix0_bzj2MiRyLu2zQMgg9uzZrC2k1wcNtGllD6ClZG71lwPoVfAdzCf3nBEqnW1SZJ7kW0RkvzVlortVrIhtBk3Hw8FdOmiTfe6XqfpYLGZDwzseDqKBVB/wQ_DXEIV47Lm62n1roWxZtdG4fRAC8ZTOI0SQwemqJEqNEgdJJqGLQzUPjxAzFHeFvT_eSfi2GSCCEAFRNe5ktdIThVce2ZRrtfwvPpppQ7iHrIMniecMRAtdEdKOxg3pc43hnFaRD_IEdD/yHncQ/_iG2QnFHeNB/areZ9vheNQ/GKtCqonmlVSS/Nx_0yRcjS_888cjp_Aw%3d%3d

https://sites.google.com/site/teachpex/Home/pex-usage-tips

Microsoft Visual Studio 2015 IntelliTest

16https://msdn.microsoft.com/en-us/library/dn823749.aspx

https://msdn.microsoft.com/en-us/library/dn823749.aspx

Microsoft Visual Studio 2015 IntelliTest

17https://msdn.microsoft.com/en-us/library/dn823749.aspx

https://msdn.microsoft.com/en-us/library/dn823749.aspx

18

Recall: An Example using IntStack


}

Three CUTs


}


}

No need to describe test data

• Generated automatically

Single PUT replaces multiple CUTs

• With reduced size of test code

public void PUT(int[] elem) {Assume.IsTrue(elem != null);IntStack stk = new IntStack();for(int i in elem)

stk.push(elem);Assert.AreEqual(elem.Length, stk.count());

}

An equivalent PUT

19

Test Generalization: CUTs PUT

Major Steps

• S1: Parameterize

• S2: Generalize Test Oracle

• S3: Add Assumptions

Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf

http://taoxie.cs.illinois.edu/publications/fase11-put.pdf

Example00: public class SettingsGroup{

01: MSS storage; ...

02: public SettingsGroup(MSS storage) {

03: this.storage = storage;

04: }

05: public void SaveSetting(string sn, object sv) {

06: object ov = storage.GetSetting( sn );

07: //Avoid change if there is no real change

08: if (ov != null ) {

09: if(ov is string && sv is string && (string)ov==(string)sv ||

10: ov is int && sv is int && (int)ov==(int)sv ||

11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||

12: ov is Enum&& sv is Enum&& ov.Equals(sv))

13: return;

14: }

15: storage.SaveSetting(sn, sv);

16: if (Changed != null)

17: Changed(this, new SettingsEventArgs(sn));

18: }}

20

SettingsGroup class of NUnit with the SaveSetting method under test

An Existing CUT00: public class SettingsGroup{

01: MSS storage; ...

02: public SettingsGroup(MSS storage) {

03: this.storage = storage;

04: }

05: public void SaveSetting(string sn, object sv) {



08: if (ov != null ) {

09: if(ov is string && sv is string && (string)ov==(string)sv ||

10: ov is int && sv is int && (int)ov==(int)sv ||

11: ov is bool&& sv is bool&& (bool)ov==(bool)sv ||

12: ov is Enum&& sv is Enum&& ov.Equals(sv))

13: return;

14: }

15: storage.SaveSetting(sn, sv);

16: if (Changed != null)

17: Changed(this, new SettingsEventArgs(sn));

18: }}

21

00: //tg is of type SettingsGroup

01: [Test]

02: public void TestSettingsGroup() {

03: tg.SaveSetting("X",5);

04: tg.SaveSetting("NAME","Tom");

05: Assert.AreEqual(5,tg.GetSetting("X"));

06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));

07: }

Existing CUT

SettingsGroup class of NUnit with the SaveSetting method under test

?

Issues with Existing CUT

22

Only CUT for verifying SaveSetting method• Does not verify the behavior for the types bool and enum

• Does not cover the true branch in Line 8

Does test generalization addresses these two issues

…



08: if (ov != null ) { …

23

S1 - Parameterize Promote all primitive values as arguments

• name of setting as a parameter of type string

• string “TOM” and int 5 as a parameter of type object

helping IntelliTest/Pex generate concrete values based on theconstraints encountered in different paths

Promote non-primitive objects such as receiver objectsas arguments

helping IntelliTest/Pex generate object states for the receiverobjects that can cover additional paths

//Original CUT



04: tg.SaveSetting("NAME","Tom");….

//New PUT

02: public void TestSave(

SettingsGroup st,

string sn, object sv){

….

24

S2 – Generalize Test Oracle

Replace the constant value, “TOM” and 5, withthe relevant parameter of the PUT

//Original CUT



04: tg.SaveSetting("NAME","Tom");

05: Assert.AreEqual(5,tg.GetSetting("X"));

06: Assert.AreEqual("Tom",tg.GetSetting("NAME"));

….

//New PUT

02: public void TestSave(SettingsGroup st,


03: st.SaveSetting(sn, sv);

04: PexAssert.AreEqual(sv,st.GetSetting(sn));

//New PUT

02: public void TestSave(SettingsGroup st,




25

S3 – Add Assumptions

IntelliTest/Pex requires guidance in generatinglegal values for the parametersE.g., Add the tag PexAssumeUnderTest (PAUT) with the parameter, i.e.,

generated value should not be null

//New PUT

02: public void TestSave([PAUT]SettingsGroup

st, [PAUT]string sn, [PAUT] object sv){



26

Example Summary

Resulting PUT

Result of S1 Parameters “sn” and “sv”

Result of S2 Line 4: “st.GetSetting(sn)”

Result of S3 Parameter “st”

00: //PAUT: PexAssumeUnderTest

01: [PexMethod]

02: public void TestSave([PAUT]SettingsGroup st,

[PAUT] string sn, [PAUT] object sv) {



05: }

27

Example - Test Generalization Results

Achieved 10% branch cov

Required 2 method calls

with values of type string and

int

2: public void

TestSave([PAUT]SettingsGroup st,

[PAUT]string sn, [PAUT] object sv){


4: PexAssert.AreEqual

(sv,st.GetSetting(sn));



4: tg.SaveSetting("NAME","Tom");….

Achieved 90% branch cov

Required only 1 method call,

sufficient to test for all types

(parameter of type object)

Original CUT New PUT

EXAMPLE PATTERNSFOR PARAMETERIZED UNIT TESTS

http://research.microsoft.com/pex/patterns.pdf


Pattern4A

• Assume, Arrange, Act, Assert

[PexMethod]void Add(List target, T value) {

PexAssume.IsNotNull(target); // assumevar count = target.Count; // arrangetarget.Add(value); // actAssert(target.Count == count + 1)//assert

}

PatternRoundtrip

• For an API f(x), f-1(f(x)) = x for all x

void ToStringParseRoundtrip(int value) {string s = value.ToString();int parsed = int.Parse(s);Assert.AreEqual(value, parsed);

}

PatternState Relation

• Observe a state change

void ContainedAfterAdd(string value) {var list = new List<string>();list.Add(value);Assert.IsTrue(list.Contains(value));

}

PatternCommutative Diagram

• Given two implementations f and g of the same function, each possible requiring a different number of steps, i.e. f(x)=f1(f2(…(fn(x)…)), and g(x)=g1(g2(… (gm(x)…)), then it should hold thatf1(f2(…(fn(x))…) = g1(g2(…(gm(x)…)) for all x.

string Multiply(string x, string y);int Multiply(int x, int y);

void CommutativeDiagram1(int x, int y) {string z1 = Multiply(x, y).ToString();string z2 = Multiply(x.ToString(), y.ToString());PexAssert.AreEqual(z1, z2);

}

Code Hunt: Turning Pex/PUT into a Coding Game

https://www.codehunt.com/

https://www.codehunt.com/

Behind the Scene of Code Hunt

Secret Implementation

class Secret {public static int Puzzle(int x) {

if (x <= 0) return 1;return x * Puzzle(x-1);

}}

Player Implementation

class Player {public static int Puzzle(int x) {

return x;}

}

class Test {public static void Driver(int x) {

if (Secret.Puzzle(x) != Player.Puzzle(x))throw new Exception(“Mismatch”);

}}

behaviorSecret Impl == Player Impl

34

It’s a game!

• iterative gameplay• adaptive• personalized• no cheating• clear winning criterion

code

test cases

Bishop et al. Code Hunt: Experience with Coding Contests at Scale. ICSE 2015, JSEEThttp://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf

http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf


Microsoft Visual Studio 2015

- IntelliTest

JUnit TheoriesPexMethod

Test GeneralizationCUTs PUTs

PUT Patterns PUTs

…

Thank You

37

https://sites.google.com/site/teachpex/http://research.microsoft.com/pexhttp://taoxie.cs.illinois.edu/

Collaboration is Welcome!

Email: [email protected]

http://research.microsoft.com/pex




Microsoft Visual Studio 2015

- IntelliTest

JUnit TheoriesPexMethod

Test GeneralizationCUTs PUTs

PUT Patterns PUTs

…

http://taoxie.cs.illinois.edu/[email protected]

https://sites.google.com/site/teachpex/


mailto:[email protected]


PatternRoundtrip

• For an API f(x), f-1(f(x)) = x for all x

void PropertyRoundtrip(string value) {var target = new Foo();target.Name = value;var roundtripped = target.Name;Assert.AreEqual(value, roundtripped);

}



PatternSanitized Roundtrip

• For an API f(x), f-1(f(f-1(x)) = f-1(x) for all x

void ParseToString(string x) {var normalized = int.Parse(x); var intermediate = normalized.ToString(); var roundtripped = int.Parse(intermediate);Assert(normalized == roundtripped);

}



PatternReachability

• Indicate which portions of a PUT should be reachable.

[PexExpectedGoals]public void InvariantAfterParsing(string input){

ComplexDataStructure x;bool success = ComplexDataStructure.TryParse(

input, out x);PexAssume.IsTrue(success);PexGoal.Reached(); x.AssertInvariant();

}



PatternRegression Tests

• Generated test asserts any observed value

– Return value, out parameters, PexGoal

• When code evolves, breaking changes in observable will be discovered

int AddTest(int a, int b) {return a + b; }

void AddTest01() {var result = AddTest(0, 0);Assert.AreEqual(0, result);

}http://research.microsoft.com/pex/patterns.pdf


PatternSame Observable Behavior

• Given two methods f(x) and g(x), and a method b(y) that observes the result or the exception behavior of a method, assert that f(x) and g(x) have same observable behavior under b, i.e. b(f(x)) = b(g(x)) for all x.

public void ConcatsBehaveTheSame(string left, string right)

{PexAssert.AreBehaviorsEqual(

() => StringFormatter.ConcatV1(left, right),() => StringFormatter.ConcatV2(left, right));

}



PatternAllowed Exception

• Allowed exception -> negative test case

[PexAllowedException(typeof(ArgumentException))]void Test(object item) {

var foo = new Foo(item) // validates item

// generated test (C#)[ExpectedException(typeof(ArgumentException))]void Test01() {

Test(null); // argument check}



45

S4 – Add Factory Method IntelliTest/Pex requires guidance handling non-

primitive objectse.g., Add factory methods that generate instances of non-primitive objects

00: //PAUT: PexAssumeUnderTest

01: //MSS: MemorySettingsStorage (class)

01: [PexFactoryMethod(typeof(MSS))]

02: public static MSS Create([PAUT] string[] sn, [PAUT]object[] sv) {

03: PexAssume.IsTrue(sn.Length == sv.Length);

04: PexAssume.IsTrue(sn.Length > 0);

05: MSS mss = new MSS();

06: for(int count = 0; count < sn.Length; count++) {

07: mss.SaveSetting(sn[count], sv[count]);

08: }

09: return mss;

10: }

• Help create different object states for MSSThummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf


46

S5 – Add Mock Object

Pex faces challenges in handling code that

interacts with external environment, e.g., file system• Write mock objects to assist Pex and to test features in isolation



47

Empirical Study

RQ1: Branch coverage• How much higher percentage of branch coverage is achieved

by retrofitted PUTs compared to existing CUTs?

RQ2: Defect detection• How many new defects (that are not detected by existing

CUTs) are detected by PUTs and vice-versa?

RQ3: Generalization effort• How much effort is required for generalizing CUTs to PUTs?



Study Setup Three Subject Applications

48

Subjects Downloads Code Under Test

# Classes # Methods KLOC

Test Code

# Classes # CUTs KLOC

NUnit 193,563 9 87 1.4 9 49 0.9

DSA 3,239 27 259 2.4 20 337 2.5

QuickGraph 7,969 56 463 6.2 9 21 1.2

TOTAL 92 809 10 38 407 4.6

High downloads count

Lines of code under test: 10 KLOC

Number of CUTs: 407Thummalapenta et al. Retrofitting Unit Tests for Parameterized Unit Testing. In FASE 2011http://taoxie.cs.illinois.edu/publications/fase11-put.pdf


Study Setup – cont.

49

Conducted by the first and second authors

• No prior knowledge of subjects

• Two years of experience with PUTs and Pex

• Retrofitted 407 CUTs (4.6 KLOC) as 224 PUTs (4.0 KLOC)

Generated three categories of CUTs• C1: Existing CUTs

• C2: CUTs generated from PUTs

• C3: Existing CUTs + RTs (tests generated using Randoop) Help show that benefits of generalization cannot be achieved by

generating tests randomly



RQ1: Branch Coverage

50

Subjects # RTs Branch Coverage

CUTs (%) CUTs + RTs (%) PUTs (%)

Overall Increase

(%)

MaximumIncrease

(%)

NUnit 144 78 78 88 10 52

DSA 615 91 91 92 1 1

QuickGraph

3628 87 88 89 2 11

CUTs + RTs: Added 144, 615, and 3628 tests using Randoop

Branch coverage increased by 0%, 0%, and 1%

CUTs generated from PUTs: Branch coverage increased by 10%, 1%, and 2%



RQ2: Defect Detection

51

RTs: CUTs generated using Randoop

Pex without generalized PUTs: CUTs generated by applying Pex on public methods

CUTs generated from PUTs: CUTs generated by applying Pex on generalized PUTs

• Detected all defects detected by the first two categories

CUTs category #Failing Tests

DSA NUnit QuickGraph

# Real Defects

Basic CUTs 0 0 0 0

RTs 90 25 738 4

Pex without generalized PUTs

23 170 17 2

CUTs generated from PUTs

15 4 0 19



RQ3: Generalization Effort

52

Both authors conducted comparable amount of generalization

Effort spent in hours

• NUnit: 2.8 hrs

• DSA: 13.8 hrs

• QuickGraph: 1.5 hrs

Effort is worthwhile compared to benefits of test generalization



next generation developer testing: parameterized testing

Spiritual