theoretical comparison of partial order reduction: java...

INOM EXAMENSARBETE TEKNIK,GRUNDNIVÅ, 15 HP

, STOCKHOLM SVERIGE 2017

Theoretical Comparison of Partial Order Reduction: Java Pathfinder vs. Moonwalker

LINN JENSEN

KTHSKOLAN FÖR DATAVETENSKAP OCH KOMMUNIKATION

Theoretical Comparison ofPartial Order Reduction: JavaPathfinder vs. Moonwalker

LINN JENSEN

Bachelor’s thesis in Computer ScienceDate: June 21, 2017Supervisor: Roberto GuancialeExaminer: Örjan EkebergSwedish title: Teoretisk Jämförelse av Partiell Order Reduktion:Java Pathfinder vs. MoonwalkerSchool of Computer Science and Communication

iii

Abstract

A problem in modern model checkers is the state explosion problem.Many methods exists to reduce the problem whereas partial order re-duction is one of the most common methods. The partial order re-duction is designed to reduce states that are not relevant for concur-rency bugs by making transitions through non-relevant states. Thereare many different ways to do the partial order reduction and manymodel checkers uses their own implementation. With different imple-mentations the outcome may differ.

This report examines the differences between two similar modelcheckers with different implementations of the partial order reduc-tions, Java Pathfinder and Moonwalker, by focusing on its state reduc-tion feature and the time it takes to do the reduction. By doing a theo-retical comparison this paper tries to answer the questions stated andthe testing of Java Pathfinder will deduce how Moonwalker wouldbehave compared to Java Pathfinder for a relevant code. The resultwas that both Moonwalker and Java Pathfinder reduces most of thestates, where Java Pathfinder outperforms Moonwalker slightly. Themore interesting result was the time it would take for Moonwalker andJava Pathfinder to do their partial order reduction where Moonwalkercould use double or even triple the time compared to Java Pathfinder.

iv

Sammanfattning

Ett problem med moderna modell kontrolleringar är det kombinato-riella explosions problemet. Det finns många metoder som kan redu-cera detta problem varav partiell order reduktion är en av de vanli-gaste metoderna. Den partiella order reduktionen är utformad för attminska tillstånd som inte är relevanta för samtidighetbuggar genomatt skapa en övergång genom icke-relevanta tillstånd. Det finns mångaolika sätt att göra en partiell order reduktion och många modell kon-trollerare använder sin egen implementation. Med olika implementa-tioner kan resultaten variera.

Den här rapporten undersöker olikheterna mellan två liknande mo-dell kontrollerare med olika implementationer av den partiella orderreduktionen, Java Pathfinder och Moonwalker, genom att fokusera påderas tillståndsreduktion och tiden det tar att göra reduktionen. Ge-nom att göra en teoretisk jämförelse kommer den här rapporten attförsöka svara på vilken partiell order reduktion är bäst och testa JavaPathfinder för ett antagande om hur Moonwalker skulle bete sig i rele-vant kod. Resultatet var att Moonwalker reducerar likvärdigt antal till-stånd som Java Pathfinder, men att Java Pathfinder överträffar Moon-walker en aning. Mer intressant var resultatet för tiden, där Moonwal-ker skulle använda mellan två och tredubblad tid jämfört med JavaPathfinder.

Contents

1 Introduction 11.1 Subject Background . . . . . . . . . . . . . . . . . . . . . . 11.2 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.3 Problem Statement . . . . . . . . . . . . . . . . . . . . . . 21.4 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background 52.1 Single-Threaded Process . . . . . . . . . . . . . . . . . . . 52.2 Multi-Threaded Process . . . . . . . . . . . . . . . . . . . 52.3 Concurrent Problems . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 Atomicity-Violation bugs . . . . . . . . . . . . . . 62.3.2 Order-Violation Bugs . . . . . . . . . . . . . . . . . 72.3.3 Deadlock Bugs . . . . . . . . . . . . . . . . . . . . 7

2.4 Model Checking . . . . . . . . . . . . . . . . . . . . . . . . 82.4.1 State Explosion . . . . . . . . . . . . . . . . . . . . 8

2.5 Java Pathfinder . . . . . . . . . . . . . . . . . . . . . . . . 92.5.1 Choice Generator for Java Pathfinder . . . . . . . 92.5.2 Partial Order Reduction for Java Pathfinder . . . . 10

2.6 Moonwalker . . . . . . . . . . . . . . . . . . . . . . . . . . 122.6.1 Partial Order Reduction for Moonwalker . . . . . 12

3 Methods 153.1 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.1.1 Theoretical Comparison . . . . . . . . . . . . . . . 153.1.2 Java Pathfinder . . . . . . . . . . . . . . . . . . . . 15

3.2 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.3 Test Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

v

vi CONTENTS

4 Results 184.1 Theoretical Comparison . . . . . . . . . . . . . . . . . . . 184.2 Test Results . . . . . . . . . . . . . . . . . . . . . . . . . . 20

4.2.1 States . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2.2 Runtime . . . . . . . . . . . . . . . . . . . . . . . . 21

5 Discussion 235.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

6 Conclusion 26

Bibliography 27

A Java Code 28A.1 Main.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28A.2 Bug.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

B Java Bytecode 31B.1 Main.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31B.2 Bug.java . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

C Test Results 36C.1 Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

C.1.1 Minimum Runtime . . . . . . . . . . . . . . . . . . 36C.1.2 Maximum Runtime . . . . . . . . . . . . . . . . . . 36C.1.3 Average Runtime . . . . . . . . . . . . . . . . . . . 37

C.2 States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

Chapter 1

Introduction

This chapter will cover the purpose of the thesis together with theproblem statement and relevant constraints.

1.1 Subject Background

It is important to verify correctness in all programs written and test-ing is therefore an important aspect of most programs. It is hard totest correctness when a program is running on multiple threads sincethe scheduling of different threads may differ each time it is executed.Since it is hard to use normal tests when verifying concurrent pro-grams model checkers are often used instead. A model checker worksby checking every possible system scenario systematically to verify ifa concurrency error is possible.[1]

Java Pathfinder and Moonwalker are two rather similar model check-ing programs used for this purpose. Java Pathfinder is a large, well-known and well used model checking tool that is used for Java byte-code. On the other hand, Moonwalker which was inspired by JavaPathfinder is used for CIL bytecode. Both of these programs are goodtools used to locate concurrent errors. As it is important to verify cor-rectness it is also important to compare the tools that are used for thatpurpose.[2][visser2005mode]

1

2 CHAPTER 1. INTRODUCTION

1.2 Purpose

One of model checkers biggest issues is the state explosion problemwhich occurs due to the interleaving of threads. For a model checkerthat verifies every possible scenario, this will create state transitions ina rapid phase for each new thread or instruction.[1]

To prevent the state explosion model checkers tries to reduce theamount of states that is necessary to look at in a concurrency perspec-tive. Therefor it is of importance to compare how this reduction ismade for both model checkers and compare it with each other.

1.3 Problem Statement

The problem statement of this report is which of Java Pathfinder andMoonwalker is better at reducing the number of states. The answer isobtained by doing a comparison between the two programs. It is doneby focusing on the reduction of states and the time the reduction willtake. The comparison is done in two phases, first a theoretical com-parison of the programs. Then by running a concurrent code in JavaPathfinder, this thesis will try to conclude how Moonwalker wouldbehave differently since it can not remove its partial order reduction..

1.4 Constraints

The report is focused on one way to reduce states used in model check-ers, the partial order reduction, and ignore any other possible reduc-tions.

Furthermore, there are many various algorithms and concurrencyproblems that could be used for testing Java Pathfinder. This studywill consider the common concurrency problem, the buying tickets is-sue. If not written properly this problem can lead to overbooking orselling the same item to multiple persons.

CHAPTER 1. INTRODUCTION 3

1.5 Terminology

Atomicity is when the serializability among memory accesses is ex-ecuted in the same order as it is stated in the code. A code-sequence is atomic if it is executed in this manner.

Cascade refers to the set of choice combinations for a transition that isassociated with multiple choice generators.

CIL is also known as Common Intermediate Language and is a plat-form independent assembly language that is run on a virtual ma-chine. The programming languages that is translated into CILbytecode is among others C#, J# and C++/CLI.

Collector Reachability refers to the garbage collector’s ability to reachan object that is still reachable due to reference from other reach-able objects or call stack.

Critical Section is the sections in a code that some of or all of the sec-tion have a memory address that is accessed by multiple proces-sors or threads.

Java Pathfinder is referring to version 7 if nothing else is stated.

Listener is waiting and listening for a certain event that triggers whatthe listener is listening for.

Moonwalker is referring to the 1.0.1 version if nothing else is stated.

Nondeterministic is referring to a code or algorithm where the out-come is not known before runtime and that may vary betweenexecution.

Object Graph is the relation between objects presented as a graph.

State saves the status of the current execution. It saves the kernel statewhich is the application’s heap and threads, it saves the trail tothe last transition and the current and next possible states thatproduces different transitions.

Strict Memory Model is when a read to a memory occur it will returnthe most recent write.

4 CHAPTER 1. INTRODUCTION

Recursive Locks means that the same thread can lock the same mutexmultiple times without chance of deadlock.

Relaxed Memory Model is when the load operations may be reorderedas long as they do not modify the behaviour of that single thread.Is used in C/C++ among other programming languages.

Thread-Local Context is the memory address space which is local toa thread.

Transition is in this report a sequence in one thread with no contextswitch that leads from one state to another. To start a new transi-tion a choice have to be made.

Chapter 2

Background

This background covers the knowledge needed for this study. It de-scribes multithreading, relevant concurrency problems, model check-ers and both of the model checking tools used in this study.

2.1 Single-Threaded Process

A single-thread process is the classic view of a process where there isonly one stack in the memory and the thread have its own addressspace. Every single-threaded process runs their code from start to fin-ish in order.[3]

2.2 Multi-Threaded Process

Multithreading is a term used in programming, the core concept isto make programs do multiple things at the same time. When a pro-gram is multithreaded it splits up the code on multiple threads thatare then run separately. This programming model is called concurrentprogramming. When multithreading, the heap and code is shared butthe stack is split up in thread-local storages, which is the local stack foreach thread. [3]

This kind of multithreading is high level and does not explain howthe code is actually executed on the machine. A multithreaded pro-gram can run on a single threaded machine, it will however not benefitfrom being multithreaded. The multiple threads in the programs can

5

6 CHAPTER 2. BACKGROUND

all run on the same thread on the hardware level but the instructionsfrom the code are scheduled to run in a specific order on the machine,ultimately turning into a single threaded program. This is also true formulticore machines where the number of threads in the program is notequal to the number of threads on the machine. This is necessary formaking multithreaded programs platform independent. This schedul-ing can be done in many different ways and can affect the behavior ofbadly written programs. Looking at the scheduling is a critical part ofmodel checking which is mentioned below.[3]

2.3 Concurrent Problems

Multithreaded programs are more complex and hence harder to debugthan regular programs. Furthermore new errors can emerge in a con-current program. Most of these errors occur when the atomicity is notmade properly. This is because all threads share heap and code and itis also possible for them to access each others stacks.[3]

If every thread in a multithreaded program was isolated and didnot depend on any other thread there would not be any problems.The programmer should strive to make the threads as independentas possible but in real world applications this is often not an option.When two threads are dependent on each other many different errorsmay occur. [3]

2.3.1 Atomicity-Violation bugs

Atomicity-violated bugs happen when the programmer assumes thatthe code is atomic inside a critical section but have failed to do so dur-ing execution. These bugs can be accounted for two thirds of all thenon-deadlock concurrency bugs. Because of this it is very importantto make sure that the bugs do not reach the surface after the programhave been deployed.[3]

When two or more threads are able to access the same part of thememory simultaneously and where one of these threads want to writeto that memory address a data-race is likely to occur. This can lead toone thread reading a variable before the other thread have been ableto write back their operation.[3]

CHAPTER 2. BACKGROUND 7

Implicit Atomicity

Some OS supports implicit atomicity meaning that the memory op-erations are grouped into chunks. Within the chunk, the processor isallowed to reorder the instructions. By doing this the performance gapbetween strict memory models and relaxed memory models will bereduced and in the meantime the hardware complexity will remainlow.[3][4]

By using Implicit atomicity it will be harder to switch between pro-cesses since it can only be made near chunk boundaries. The softwarewill not be able to know about when and how the implicit atomicityworks. [3]

2.3.2 Order-Violation Bugs

The order-violation bugs appear when the order desired between twothreads are flipped. This can lead to crashes when a thread tries toaccess a memory thinking it has been initialized when in reality it hasnot.[3]

2.3.3 Deadlock Bugs

A deadlock occur when two or more threads hold a lock that the otherthreads needs. The conditions for deadlock is mutual exclusion, hold-and-wait, no preemption and circular wait. A mutual exclusion ap-pears when two or more threads hold a resource that another threadneeds. The hold-and-wait condition appears when the threads con-tinue holding a resource that is needed while it waits for the resourcethey need themselves. The no preemption condition appears when thethreads are not able to forcefully take resources and the circular waittells that all threads in the deadlock waits for resources in a circle.[3]

If the program contain less than four of these condition, a deadlockmay not occur.[3]


2.4 Model Checking

Model checking is a formal method for verification. This means thatit uses mathematics for modeling and analyzing with the aim to es-tablish system correctness. This is done by checking every possiblesystem scenario systematically.

Model checking uses three different phases: modeling, runningand analysis phase. Where the program first model every possibleoutcome and then run them so that it can be analyzed afterwards.

It is important to use formal methods when verifying system cor-rectness since failure is unacceptable in many different systems thatare used today. Some failures could even lead to dangerous situations.

Other possible ways to validate systems are simulations, testingand deductive verification. When simulating or testing one has to ex-periment to be sure that the system is correct. Deductive verificationuses axioms and proof rules to prove the correctness. While deductiveverification is widely recognized it is very time-consuming and canonly be used by those who have studied this.[1]

2.4.1 State Explosion

Model checking is conceptually very time consuming and scales badly.One of the challenges with the model checking is the state explosionwhich occurs in a system where many components interact with eachother or when the system have data structures that can take many dif-ferent values. This is due to the interleaving of threads. In modelcheckers used today it is possible to handle spaces with around 108

to 109 states without problem, but with tailored data structures evenlarger states are possible.[1][5][6]

Partial Order Reduction

A naive implementation of model checking would not be very usefulbecause it would not be able to test any decent sized programs. Thatwould take too long. In order for model checkers to be useful for ver-ification it needs to be able to test programs which tend to be long


and complex since it is these kinds of programs that are needed to beverified for correctness. To achieve this model checkers employ manydifferent techniques to optimize the runtime.[6]

One approach to this is partial order reduction (POR) which iden-tifies if a state is thread independent or not. If a state is independentthen it is not necessary to run all possible outcomes of that state everysingle iteration. This is done by skipping all transitions where concur-rent errors may not apply. By using partial order reduction a modelchecker can minimize the amount of states that needs to be visited ineach iteration and thus the state explosion will not grow as rapidly.When partial order reduction is implemented more than 70% of thestates can be reduced.[6][7]

2.5 Java Pathfinder

Java Pathfinder (JPF) is an open source model checker developed byNASA that is used for Java bytecode programs specified in explicitstate software. JPF was released in 1999 as a Java-to-Pomela translator,but is now running the eight version that was released in 2014. Thisstudy will focus on the seventh version since it can disable the partialorder reduction[8].

JPF is able to detect concurrency related and Java runtime relatedbugs. The concurrency related bugs are among others deadlocks andmissed signals while Java runtime related bugs are unhandled excep-tions or heap usage. JPF is also able to find complex application spe-cific assertion bugs.[8]

JPF uses a variety of mechanisms in order to fight the main chal-lenge which is scalability. The program uses user extensible state ab-straction and matching, configurable search strategies, user definableheuristics and on-the-fly partial order reduction.[8]

2.5.1 Choice Generator for Java Pathfinder

In Java Pathfinder, the choice generator is working together with thepartial order reduction and is such important to understand.


A choice generators main purpose is to do the right choices for themodel checker to be able to reach a state that is considered interesting.In Java Pathfinder the choice generator is required to be decoupled,meaning the thread choices should be independent of data choices.It is also required to encapsulate sets and enumeration for dedicatedtype specific objects so that the virtual machine only knows the mostbasic types. A third requirement is that the parametrization should bepossible at runtime together with the selection of classes that representheuristics. [9]

The JPF choice generator has in addition to data choices an abilityto change the scheduling choices that is not controlled by the test ap-plication. [9]

A choice generator is implemented with two parts. The first one iscalled top half and creates a new choice generator and registers it. Asa new choice generator is made the transition is ended to be able tostart a new transition. The second part is called bottom half and it isthis one that does all the work. It processes the choice that has beenmade. [9]

When using cascaded choice generators it is required more thanstated before. That is since the virtual machine might otherwise createa thread choice generator for scheduling choices. This happens when aon-the-fly partial order reduction is implemented which is the case inJPF. Since multiple listeners are able to use different choice generatorsfor different purposes for the same type of choice generator. To solvethis JPF uses a unique identifier for each choice generator so that mul-tiple choice generators of same type can exist at the same time. To seeif the system is cascaded it only checks if there is already a registeredchoice generator, if it does it sets a cascaded attribute for it.[9]

2.5.2 Partial Order Reduction for Java Pathfinder

JPF uses on-the-fly partial order reduction which automatically deter-mines if the next instruction is scheduling relevant or yields nonde-terministic results during runtime. By doing so JPF does not have tothink about static analysis or user instrumentation. [7]


When partial order reduction is enabled in JPF it sends a requestto the virtual machine that executes all the instructions in the currentthread until either the next instruction is scheduling relevant or yieldsa nondeterministic result. The request is delegated to the instructionobject itself. It creates a choice generator if the instruction is a transi-tion breaker and is then re-executed. [7]

To determine the scheduling relevance JPF looks at a couple of dif-ferent factors. One factor is the different instruction types, where ap-proximately 10% of Java bytecode instructions are scheduling relevant.These 10% are direct synchronization instructions, field or array ele-ment access and object methods. Another factor is object reachabilitywhich verifies which field accesses that are reachable from two or morethreads. This task is similar to a task that is performed during garbagecollection and is thus extended from garbage collection to support par-tial order reduction. A third factor, the threads and lock informationmake sure that JPF does not have to break the transition if the threadis schedule relevant or object reachability if there is only one threadrunning at that time. [7]

While JPF traverses through the instructions with these factors itsets an attribute for each instruction if they are either never relevant,always scheduling relevant, only relevant in the context of other runna-bles or if it is only relevant for toplevel locks. It also uses a listener torequest a rescheduling if a instruction execution notification is given.The JPF also have a verify class that communicates between the testapplication and JPF to control thread interleaving. [7]

The JPF uses its garbage collectors object traversal which makesthe partial order reduction reachability a subset of the garbage collec-tor reachability. The garbage collector will see that all threads withintheir threadgroup are mutually reachable. To calculate the partial or-der reduction reachability it is divided into two phases. The first phasewill mark all objects of the root set non-recursively. It will record theid of the reference thread and in case of a static field being reachablefor an object or other thread it will set the objects status to shared.The second phase will traverse all heap objects recursively, trying tofind an object that is already referenced by another thread, if this is


the case then set that object to shared. JPF also looks for lock protec-tions to make sure if a field access is scheduling relevant or not. Itrecords the set of locks held when executing, computing the set inter-section at subsequent accesses and if the set does not become emptyfor a configurable number of field accesses then the field is marked aslock protected. After it has been marked as lock protected, if the set isthen empty it will give out a warning and make the field a transitionbreaker again.[7]

Most of the unwanted transition breakers that remain is due to thatJPF does not consider accessibility and that a write once, read multiplecondition may be detected to early for fields that are inside immutableobjects or are not final.[7]

2.6 Moonwalker

Moonwalker is a model checker inspired by JPF’s virtual machine butinstead of verifying Java bytecode it verifies CIL bytecode programs.But even though Moonwalker is inspired by JPF, it has its own design,implementations and organisation of classes and algorithms that aredifferent from JPF. Moonwalker was developed and released in Uni-veristy of Twente as a MSc thesis in August 2006 and was then calledMMC, Mono model checker, and the current version was released inApril 2008 and was with that release renamed Moonwalker.

Moonwalker iterates over the state of CIL bytecode programs dur-ing runtime just as JPF does and is also able to detect concurrencyrelated and runtime related bugs.

2.6.1 Partial Order Reduction for Moonwalker

Moonwalker uses two different partial order reduction techniques, thedynamic partial order reduction and partial order reduction using ob-ject escape analysis, to be able to group more instructions together.[2]

In Moonwalker the partial order reduction will start with goingthrough the object escape analysis and then if it does not reveal onlyon possible traversal then it will run dynamic partial order reductionthrough all possible transitions.[2]


Object Escape Analysis

Object escape analysis analyses the objects in the program by watch-ing the object graph and see where an object is reachable from mul-tiple threads or are in their own relaxed memory models. It doesso by setting an attribute for each object. If the object is marked asUNMARKED, which every object is first initialized to, it has not yetbeen reached from the callstack, and might not ever. Then while thecallstack traverse it marks all the unmarked objects with their threadidentifier. When the callstack traverse to an object with a thread iden-tifier which is not the same as the current thread, it will set the objectto SHARED, otherwise if it has the same thread identifier it will leaveit as it is. [2]

When this is done a recursive traverse through the objects is made.If a child object have a different propagated attribute than the objectthat is recursivly traversed or if the child’s parent is set as shared thenthe child will also get marked as shared. After the recursive traversethe garbage collector can come and get all objects still marked as un-marked. [2]

Next thing to do is set the persistent set. The persistent set containseither any transition that has access to a thread unshared object or thewhole enabled set. If the set only contains a transition then it is calleda singleton persistent set and it can be merged with the transitions pre-decessor. This is because the persistent set is known to only have onesuccessor state if it is a singleton. [2]

The object analysis in Moonwalker is working with the garbagecollector to reduce heap symmetry.[2]

Dynamic Partial Order Reduction

The dynamic partial order reduction assumes that the current traversedset of states is the only one that is a possibility for each new state that isvisited afterwards. When the state is further explored it is determinedif there are any dependencies between the explored transitions that arefound on the backtrack stack and the subsequent set. The explorationof a state is done by looking at its backtrack stack and trying to find de-


pendencies between transitions. The transitions are dependent if theyaccess the same field entity. If there is no other transition accessing itthat lies in between then the state is shared.[2]

Chapter 3

Methods

In this chapter the methods for this study will be described. Initiallythe chapter will describe the approach and later the algorithm used inthe study and the test cases chosen.

3.1 Approach

The method is divided into two parts. The first part is the theoreticalcomparison and the second is testing.

3.1.1 Theoretical Comparison

For the theoretical comparison this study will use the information pre-sented in the background as a ground for comparison to find a conclu-sion about the differences of partial order reduction for two differentmodel checkers, Java Pathfinder and Moonwalker and how they work.For a model checker it is highly important to keep the number of statesto a minimum to reduce the time it takes to complete the verification.The theoretical comparison will because of this focus on the numberof states the model checker can reduce since this is a key aspect to re-ducing the running time.

3.1.2 Java Pathfinder

After the theoretical comparison this study will run a few test cases foran algorithm to see if the conclusion made from the theoretical com-parison holds. To get this result this study will run all test cases with

15

16 CHAPTER 3. METHODS

and without the partial order reduction. Since the state explosion willbe much greater without the partial order reduction these test casescan not be too big since Java Pathfinder would not be able to managethat.

3.2 Algorithm

The algorithm used is implemented by Amit Rotstein and taken fromthe Brigham young university CS department in Utah and is one ofmultiple algorithms used to compare concurrency tools from that uni-versity.

The algorithm, seen in appendix A, that is used, is a ticket machinewhere there is a demand to buy a higher amount of tickets than avail-able, which makes a datarace occur. The input is the amount of buyerstrying to get a ticket and the amount of buyers that should not be ableto get a ticket. Then a thread for each buyer is set up and the variablefor number of tickets sold is incremented. The datarace occurs sincethere are no locks between the check if the number of tickets that havebeen sold has reached the maximum capacity and the increment of thesame sold tickets.

3.3 Test Cases

This report will test the algorithm with four different input parametersto see how the number of states that Java Pathfinder uses and thus areable to reduce. It will also see how much time the test cases run. Thetest cases will see how Java Pathfinder’s states and runtime escalateswith the number of threads.

The four different inputs will all have the same amount of buyersthat will not be able to buy a ticket but will use a different amount ofbuyers which means a different amount of threads. All tests will runten times and afterwards the minimum, maximum and average timewill be saved along with the number of states for that test. Since thenumber of states reduced will be deterministic it is not necessary totake the average.

CHAPTER 3. METHODS 17

The inputs used in the test cases are 4, 5, 6 and 7 buyers, or threads,where 3 of them are not able to buy a ticket.

Chapter 4

Results

This section will describe and explain the result from both the theoret-ical comparison and the test cases with Java Pathfinder.

4.1 Theoretical Comparison

The partial order reduction is a great part of the reduction of statesand thus makes the model checker perform is due to model checkersnot being able to reduce all states. This theoretical comparison willfirst focus on the major differences between Moonwalker’s and JavaPathfinder’s partial order reduction. Secondly, it will focus on thedifferences in performance of the partial order reduction. Lastly thetheoretical comparison will focus on how the different partial orderreductions will change the performance of the verification.

Even though the semantics of Java Pathfinder’s and Moonwalker’spartial order reductions are rather similar, there are two major differ-ences that may lead to different amount of states. One of these is theability in Java Pathfinder that changes the attribute to four differentscheduling levels may be chosen from compared to Moonwalker thathas two, the SHARED and the thread identifier. These four schedulinglevels are as stated in 3.5.2 if a state is never relevant, always relevant,only relevant in some transitions and only relevant of toplevel locks.By using multiple attributes Java Pathfinder is able to reduce somestates depending on the transition which is checked. Moonwalker israther good at this too but will set a state to SHARED in all transitionsif it in one transition is shared.

18

CHAPTER 4. RESULTS 19

The second rather miniscule difference is when a program usesrecursive locks. Both Moonwalker and Java Pathfinder use a tran-sition break when reaching a lock. But if the lock is called recursivelythen Java Pathfinder will only break on the first recursion while Moon-walker will break on every recursive call.

When running the partial order reduction on Moonwalker manyrecursions and backtrackings are made to reduce the amount of states.When the object escape analysis is running it traverses through allrunnable states atleast once each. For each object marked as SHAREDthere will take even more time to run the whole program since it re-cursively checks all objects in a stack. After the order escape analysisMoonwalker will run the dynamic partial order reduction which willbe at runtime but will for each state explore all states that have earlierbeen checked in the current state traversal.

With Java Pathfinders on-the-fly partial order reduction it checksevery possible state during runtime of the model checker without back-tracking or checking every state recursively. When Java Pathfinder re-duces an instruction than can not be scheduling relevant, it will notbreak on them during every new transition. Neither will that code se-quence be checked again in all other possible scenarios.

The outcome from the partial order reduction would be rather sim-ilar in Java Pathfinder and Moonwalker when looking at the reductionof states. There might be a difference if there were multiple scenar-ios where two or more threads work differently but creates the samescenario. Then Java Pathfinder will be able to reduce states that arepossible to reduce in one scenario but might not be in another com-pared to Moonwalker that should not be able to have it reduced. AlsoJava Pathfinder might reduce plenty of states if a lock is claimed recur-sively. For each time the lock is claimed, except the first time, it willnot set a transition break. In Moonwalker this is not a feature and willtherefore reduce fewer states if the code is as stated.

Disregarded of what different states Java Pathfinder is able to re-duce compared to Moonwalker, the time it will take to run their dif-ferent partial order reductions will vary more than the states. To start

20 CHAPTER 4. RESULTS

with is an example of an extreme case where there is no occurring con-currency. Then the time it would take for Java Pathfinder to do the par-tial order reduction will grow at around the same pace as the growthof possible states. Meanwhile Moonwalker would take around doublethe time for each new state. That is since Moonwalker first runs theobject escape analysis which iterates through all states and then puteverything with a thread identifier on a stack that it also goes through.It will not do the dynamic partial order reduction if there is no sharedobject.

For each new possible shared object the Java Pathfinder will nottake any more time compared to when running only single threadedinstructions. That is since it analyses every next state by looking at theinstruction type and what threads that may potentially access it. It willalso analyze what threads have already been accessing the object. Thatis done once for each possible state regardless of concurrent threads ornot.

Compared to Java Pathfinder which regardless of the instructiontypes will run the same amount of time, Moonwalker’s partial orderreduction will grow more rapidly for each new shared state. Look-ing only at the object escape analysis it runs in the beginning it willtake between double and triple the time for each new state in theprogram. After that the dynamic partial order reduction will look atevery shared state and backtrack through all earlier state in that sce-nario. That makes the program, depending on how it is built, traversethrough all states, that are accessible when running the code, betweentwo and three times. This means that the partial order reduction forMoonwalker will take two to three times the time it would take to justiterate once through the states.

4.2 Test Results

The test results are divided into two parts, the first part which looks atthe runtime difference for Java Pathfinder with partial order reduction.The second part looks at how well Java Pathfinder reduces the amountof states. The exact test results can be found in appendix C.

CHAPTER 4. RESULTS 21

4.2.1 States

Figure 4.1 shows the number of states that the model cheer works withwhen partial order reduction is enabled or disabled. As seen in the fig-ure the amount of states are almost growing in double pace when JavaPathfinder does not use partial order reduction compared to when itis used. In all iterations the number of states stayed the same.

5 6 7

0

1

2

3

4

5·107

Threads

Stat

es

With Partial Order ReductionWithout Partial Order Reduction

Figure 4.1: Graph showing number of states

4.2.2 Runtime

Figure 4.2 shows the difference calculated when Java Pathfinder havepartial order reduction enabled and disabled. The reason for this is toonly focus on the impact the partial order reduction have on the result.The figure focuses on when the code uses five, six and seven threads.As seen in the graph the state explosion is most visible between thesixth and seventh thread.

22 CHAPTER 4. RESULTS

5 6 7

0

1,000

2,000

3,000

4,000

5,000

Threads

Tim

e(s

)

Minimum timeAverage timeMaximum time

Figure 4.2: Graph showing average time

Chapter 5

Discussion

This chapter will discuss the results presented in chapter 4. It will fo-cus on three main aspects. The first aspect to discuss is how the testcases results is compared to the result in the theoretical comparison.The next aspect to discuss is how Moonwalker would work if it ranthe same test case as Java Pathfinder. The last aspect will discuss whatthe results in this study means and what that could lead to.

In the theoretical comparison it was shown that Java Pathfinder isable to reduce all states that are never able to be scheduling relevant. Itis also able to reduce states partially by giving them certain attributes.If looking at the Java bytecode for the code used for the tests whichis shown in appendix B, it shows that 52 out of the 120 instructionsmight be scheduling relevant just by its instruction type. Meanwhileother reductions are possible due to either being partially relevant orin a recursive lock. As seen on Figure 4.1, the number of states re-duced is almost half when using the on-the-fly partial order reductionmethod. What is most important in the code used for testing is the runfunction. That is since the more threads are created the more times thatfunction will run. For the run function there are seven out of twelveinstructions that may be scheduling relevant, and most likely are. Thatmeans that around 41% of the instructions in this function will be re-duced which is around the same amount as in Figure 4.1. The partialorder reduction reduced almost 46% of the states which means that the5% left are reduced in the other functions.

In Figure 4.2, the one comparing time, the results are as believed in

23

24 CHAPTER 5. DISCUSSION

the theoretical comparison. For each new state it would take aroundthe same time independent of sharing or not. If one compares Figure4.1 and Figure 4.2 it is easy to see that they look rather similar whenit comes to the growth. That means that for every new state the timewill be increased linearly as stated in the theoretical comparison.

If Moonwalker would have ran the same type of code as Java Pathfi-nder, but with CIL bytecode instead of java bytecode, the number ofstates would have been rather similar. That is because of this code doesnot contain many states where the attribute is set on only relevant inthe context of other runnables or where there is a recursive lock.

The big difference between the result of Java Pathfinders tests andhow Moonwalker would perform is the time. It would take almostthree times as much time to do the reduction for Moonwalker thanJava Pathfinder. That does not mean that the difference in time suchas the one shown in Figure 4.2 would be three times larger since thegraph shows the time to reduce compared to the time not to reducebut working with multiple states.

The results here are rather clear that when it comes to reducing thenumber of states, both Java Pathfinder and Moonwalker are able to re-duce most cases. Java Pathfinder has taken it a step further by usingmore attributes than Moonwalker but since these attributes were notused in this study that would not matter for the tests. There is poten-tial for Moonwalker to improve by letting its partial order reductionchange to an on-the-fly like Java Pathfinder. That would lead to lesstime to run the partial order reduction and reduce a few more states.As stated in the background Java Pathfinder does not consider acces-sibility which may be a way for it to improve in the future. OtherwiseJava Pathfinder have found a rather good partial order reduction thatis fast and finds a lot of the potential states.

5.1 Limitations

A limitation that was discovered in this study is the inability for JavaPathfinder to disable the partial order reduction in its latest version,version 8. Because of that limitation this study have been focused on

CHAPTER 5. DISCUSSION 25

version 7 of Java Pathfinder, which have the same concept when itcomes to the partial order reduction.

Chapter 6

Conclusion

The partial order reduction of both Java Pathfinder and Moonwalkerwork in different ways, but they are in many cases able to reduce ap-proximately the same amount of states. With Java Pathfinder doing apartial order reduction which looks at every next state, Moonwalkerchecks every previous state for each state which makes it slower in away that, shown in Java Pathfinder, is not necessary. It is also shownin this report that even with a state reduction, both Moonwalker andJava Pathfinder still have a problem with the state explosion problem.

The results in this thesis may vary depending on what version ofthe model checkers that are used since other parts of the model check-ers may have been improved.

26

Bibliography

[1] Christel Baier, Joost-Pieter Katoen, and Kim Guldstrand Larsen.Principles of model checking. MIT press, 2008.

[2] Viet Yen Nguyen. “Optimising techniques for model checkers”.MA thesis. University of Twente, 2007.

[3] Remzi H Arpaci-Dusseau and Andrea C Arpaci-Dusseau. Operat-ing systems: Three easy pieces. Vol. 151. Arpaci-Dusseau Books Wis-consin, 2014.

[4] Brandon Lucia et al. “Atom-aid: Detecting and surviving atom-icity violations”. In: Computer Architecture, 2008. ISCA’08. 35th In-ternational Symposium on. IEEE. 2008, pp. 277–288.

[5] Antti Valmari. “The state explosion problem”. In: Lectures on Petrinets I: Basic models (1998), pp. 429–528.

[6] Edmund M Clarke, Orna Grumberg, and Doron Peled. Model check-ing. MIT press, 1999.

[7] On-the-fly Partial Order Reduction. 2007. URL: https://babelfish.arc.nasa.gov/trac/jpf/wiki/devel/partial_order_reduction (visited on 04/29/2017).

[8] Willem Visser and Peter Mehlitz. “Model checking programs withJava PathFinder”. In: SPIN. Vol. 3639. 2005, p. 27.

[9] ChoiceGenerators. 2007. URL: https://babelfish.arc.nasa.gov/trac/jpf/wiki/devel/choicegenerator (visited on04/29/2017).

27

Appendix A

Java Code

A.1 Main.java

/∗ ∗∗ C r e a t e d by I n t e l l i J IDEA .∗ User : amit r o t s t e i n I .D: 037698867∗ Date : Oct 17 , 2003∗ Time : 1 : 0 2 : 1 3 PM∗ /

public c l a s s Main {public s t a t i c void main ( S t r i n g [ ] args ) {

i n t numberThreads = 1 0 ;i n t cushion = 3 ;

i f ( args . length < 2) {System . out . p r i n t l n ( "ERROR: Expected 2 parameters " ) ;

} e lse {args [ 0 ] = args [ 0 ] . tr im ( ) ;args [ 1 ] = args [ 1 ] . tr im ( ) ;

numberThreads = I n t e g e r . p a r s e I n t ( args [ 0 ] ) ;cushion = I n t e g e r . p a r s e I n t ( args [ 1 ] ) ;new Bug ( " t e s t " , numberThreads , cushion ) ;

}}

}

28

APPENDIX A. JAVA CODE 29

A.2 Bug.java

import j ava . io . FileOutputStream ;import j ava . io . FileNotFoundException ;import j ava . io . IOException ;

/∗ ∗∗ C r e a t e d by I n t e l l i J IDEA .∗ User : amit r o t s t e i n I .D: 037698867∗ Date : Oct 17 , 2003∗ Time : 1 : 0 2 : 1 3 PM∗ /

public c l a s s Bug implements Runnable {s t a t i c i n t Num_Of_Seats_Sold = 0 ;i n t Maximum_Capacity , Num_of_tickets_issued ;boolean StopSales = f a l s e ;Thread threadArr [ ] ;FileOutputStream output ;private S t r i n g fileName ;

public Bug ( S t r i n g fileName , i n t s ize , i n t cushion ) {t h i s . fileName = fileName ;Num_of_tickets_issued = s i z e ;Maximum_Capacity = Num_of_tickets_issued − cushion ;threadArr = new Thread [ Num_of_tickets_issued ] ;

for ( i n t i = 0 ; i < Num_of_tickets_issued ; i ++) {threadArr [ i ] = new Thread ( t h i s ) ;

i f ( S topSales ) {Num_Of_Seats_Sold−−;break ;

}

threadArr [ i ] . s t a r t ( ) ;}

i f ( Num_Of_Seats_Sold > Maximum_Capacity ) {

30 APPENDIX A. JAVA CODE

throw new RuntimeException ( " bug found " ) ;}

}

public void run ( ) {Num_Of_Seats_Sold ++;

i f ( Num_Of_Seats_Sold > Maximum_Capacity ) {S topSales = t rue ;

}}

}

Appendix B

Java Bytecode

B.1 Main.java

Compiled from " Main . java "public c l a s s Main {

public Main ( ) ;Code :

0 : aload_01 : invoke spec ia l #1 / /4 : return

public s t a t i c void main ( java . lang . S t r i n g [ ] ) ;Code :

0 : bipush 102 : i s t o r e _ 13 : i c o n s t _ 34 : i s t o r e _ 25 : aload_06 : array length7 : i c o n s t _ 28 : i f_icmpge 22

1 1 : g e t s t a t i c #21 4 : ldc #31 6 : i n v o k e v i r t u a l #41 9 : goto 662 2 : aload_02 3 : i c o n s t _ 0

31

32 APPENDIX B. JAVA BYTECODE

2 4 : aload_02 5 : i c o n s t _ 02 6 : aaload2 7 : i n v o k e v i r t u a l #53 0 : a a s t o r e3 1 : aload_03 2 : i c o n s t _ 13 3 : aload_03 4 : i c o n s t _ 13 5 : aaload3 6 : i n v o k e v i r t u a l #53 9 : a a s t o r e4 0 : aload_04 1 : i c o n s t _ 04 2 : aaload4 3 : i n v o k e s t a t i c #64 6 : i s t o r e _ 14 7 : aload_04 8 : i c o n s t _ 14 9 : aaload5 0 : i n v o k e s t a t i c #65 3 : i s t o r e _ 25 4 : new #75 7 : dup5 8 : ldc #86 0 : i load_16 1 : i load_26 2 : invokespec ia l #96 5 : pop6 6 : return

}

APPENDIX B. JAVA BYTECODE 33

B.2 Bug.java

Compiled from "Bug . java "public c l a s s Bug implements j ava . lang . Runnable {

s t a t i c i n t Num_Of_Seats_Sold ;i n t Maximum_Capacity ;i n t Num_of_tickets_issued ;boolean StopSales ;j ava . lang . Thread [ ] threadArr ;j ava . io . FileOutputStream output ;private j ava . lang . S t r i n g fileName ;

public Bug ( java . lang . Str ing , int , i n t ) ;Code :

0 : aload_01 : invoke spec ia l #14 : aload_05 : i c o n s t _ 06 : p u t f i e l d #29 : aload_0

1 0 : aload_11 1 : p u t f i e l d #31 4 : aload_01 5 : i load_21 6 : p u t f i e l d #41 9 : aload_02 0 : aload_02 1 : g e t f i e l d #42 4 : i load_32 5 : isub2 6 : p u t f i e l d #52 9 : aload_03 0 : aload_03 1 : g e t f i e l d #43 4 : anewarray #63 7 : p u t f i e l d #74 0 : i c o n s t _ 04 1 : i s t o r e 4

34 APPENDIX B. JAVA BYTECODE

4 3 : i l o a d 44 5 : aload_04 6 : g e t f i e l d #44 9 : i f_icmpge 1015 2 : aload_05 3 : g e t f i e l d #75 6 : i l o a d 45 8 : new #66 1 : dup6 2 : aload_06 3 : invokespec ia l #86 6 : a a s t o r e6 7 : aload_06 8 : g e t f i e l d #27 1 : i f e q 857 4 : g e t s t a t i c #97 7 : i c o n s t _ 17 8 : isub7 9 : p u t s t a t i c #98 2 : goto 1018 5 : aload_08 6 : g e t f i e l d #78 9 : i l o a d 49 1 : aaload9 2 : i n v o k e v i r t u a l #109 5 : i i n c 4 , 19 8 : goto 43

1 0 1 : g e t s t a t i c #91 0 4 : aload_01 0 5 : g e t f i e l d #51 0 8 : i f _ i c m p l e 1211 1 1 : new #111 1 4 : dup1 1 5 : ldc #121 1 7 : invoke spec ia l #131 2 0 : athrow1 2 1 : return

public void run ( ) ;

APPENDIX B. JAVA BYTECODE 35

Code :0 : g e t s t a t i c #93 : i c o n s t _ 14 : iadd5 : p u t s t a t i c #98 : g e t s t a t i c #9

1 1 : aload_01 2 : g e t f i e l d #51 5 : i f _ i c m p l e 231 8 : aload_01 9 : i c o n s t _ 12 0 : p u t f i e l d #22 3 : return

s t a t i c { } ;Code :

0 : i c o n s t _ 01 : p u t s t a t i c #94 : return

}

Appendix C

Test Results

C.1 Runtime

C.1.1 Minimum Runtime

Table C.1: Minimal runtimeInput Parameters POR enabled POR disabled Difference

5 & 3 00:00:07 00:00:16 00:00:096 & 3 00:02:27 00:05:53 00:03:267 & 3 01:07:52 02:08:24 01:00:32

C.1.2 Maximum Runtime

Table C.2: Maximal runtimeInput Parameters POR enabled POR disabled Difference

5 & 3 00:00:08 00:00:18 00:00:106 & 3 00:02:56 00:06:53 00:03:577 & 3 01:11:58 02:34:41 01:22:43

36

APPENDIX C. TEST RESULTS 37

C.1.3 Average Runtime

Table C.3: Average runtimeInput Parameters POR enabled POR disabled Difference

5 & 3 00:00:07,5 00:00:17,7 00:00:10,26 & 3 00:02:44,8 00:06:12,9 00:03:28,17 & 3 01:09:37,6 02:16:50,8 01:07:13,2

C.2 States

Table C.4: StatesInput Parameters POR enabled POR disabled Difference

5 & 3 71 712 129 989 58 2776 & 3 1 240 227 2 397 359 1 157 1327 & 3 25 562 515 46 974 976 21 412 461

www.kth.se

theoretical comparison of partial order reduction: java...

Documents