QuickCheck@Neilfest(or what I learned from Neil about software testing)

John HughesChalmers University/Quviq AB

• Company founded May 2006– Tools for testing software– Profitable in the first year!

• Customers include…

Soon TBA

QuickCheck

• Tests software automatically against a formal specification

prop_reverse() -> ?FORALL({Xs,Ys},

{list(int()),list(int())}, reverse(Xs++Ys) ==

reverse(Xs)++reverse(Ys)).

Testing…

19> eqc:quickcheck(ex:prop_reverse())..........Failed! After 10 tests.{[2],[3,2]}Shrinking....(4 times){[0],[1]}false

QuickCheck

• Tests software automatically against a formal specification

prop_reverse() -> ?FORALL({Xs,Ys},

{list(int()),list(int())}, reverse(Xs++Ys) ==

reverse(Xs)++reverse(Ys)).

QuickCheck in Brief

Specification Test case Test outcome

Generate Execute

Simplify

Minimal counter- example

e.g. {[0],[1]}

A More Realistic Test Case

• A test case for the Erlang process registry

test_register() -> Pid = spawn(), register(name,Pid), Pid2 = whereis(name), assert(Pid==Pid2), unregister(name).

Spawn a new process (set up test data)

Register it—a side-effect

Inspect the results

Did the test pass?Restore the state, ready for next test

Industrial Test Cases

• A sequence of– Calls to an API under test– Checks on the results

• Not much like {[0],[1]}…

• How can we convert this kind of test case into a logical property?

QuickCheck in Industry

Specification Test case: a program

Test outcome

Generate Execute

Simplify

Minimal program

provoking a fault

Programs as Data Objects

• Why?– So failing test cases can be printed– So failing test cases can be repeated– So failing test cases can be simplified

Can we mix generation and execution?

• Randomly generate and perform each call– Save a test case as a list of functions and actual

parameters?– NO!!!

• When we repeat the test, Pid has a different value– We must use a symbolic variable!

test_register() -> Pid = spawn(), register(name,Pid),…

Test Case Language

• Do we need conditionals?

• Tests should be deterministic– Check the right branch, don’t test

• Users need simplicity—straight-line code is simple, and suffices

X = foo(…),case p(X) of true -> baz(X); false -> bar(X)end

X = foo(…),assert not(p(X)),bar(X)

Generating Test Cases

• How do we know– Which commands are valid at each point?– What test data is available at each point?– What results are expected?

• Track a test case state– Check preconditions before generating each

command– Store available data in the state– Check postconditions wrt test case state

Process Registry State

type state() = record pids::list(pid()), regs::list({atom(),pid()})end.

Available process ids

Currently registered processes

Process Registry State

• Two-level programs!– Static = test case generation time– Dynamic = test execution time

type state() = record pids::list(symbolic(pid())), regs::list({atom(),symbolic(pid())})end.

Available process ids

Currently registered processes

Two-level State Machines

• Preconditions– Checked during generation, purely static

• Postconditions– Checked during execution, purely dynamic

• Next state function– Used at both times, two-level

• Command generator– Used during generation, two-level

Process Registry

• spawn()– Adds a (dynamic) pid to the state

• unregister(Name)– Pre: Name must be registered– Removes Name from the state

• register(Name,Pid)– Adds {Name,Pid} to the state– Post: exception if Name or Pid already registered

A Failing Test Case

[{set,{var,2},{call,…,spawn,[]}}, {set,{var,3},{call,…,register,[a,{var,2}]}}, {set,{var,6},{call,…,register,[b,{var,2}]}}, {set,{var,8},{call,…,spawn,[]}}, {set,{var,9},{call,…,register,[b,{var,8}]}}]

V2=spawn(),register(a,V2),register(b,V2),V8=spawn(),register(b,V8)

Code Spec

Inconsistency!

A Buggy Spec

• spawn()– Adds a (dynamic) pid to the state

• unregister(Name)– Pre: must be registered– Removes Name from the state

• register(Name,Pid)– Adds {Name,Pid} to the state– Post: exception if Name or Pid already registered

…provided there is no exception!

”The Trick”

• What if we know part of the structure?– Check the known part in the postcondition– Add the known part statically to the state

• ”The Trick” is as useful as ever!

[{set,{var,2},{call,…,spawn,[]}}, …]

Added to the state Purely dynamic

Test Case Generator

Two-level languages are helping to find thorny bugs in industrial systems!

Randomized Generating Extension

=

Best Bug!• In Ericsson’s Media Proxy (Multimedia IP-

telephony product, SIP)

• A serious bug• Obtained from a simple specification, by

simplifying 160-command sequence!

Add Add Sub Add Sub Add Sub