Prioritising Model-Based Debugging Diagnostic ReportsWolfgang Mayer† and Rui Abreu‡ and Markus Stumptner† and Arjan J.C. van Gemund‡

†Advanced Computing Research CentreUniversity of South Australia

Australia{mayer,mst}@cs.unisa.edu.au

‡Embedded Software LabDelft University of Technology

The Netherlands{r.f.abreu,a.j.c.vangemund}@tudelft.nl

AbstractModel-based debugging has proved successful as atool to guide automated debugging efforts, but thetechnique may suffer from large result sets in prac-tice, since no means to rank or discriminate betweenthe returned candidate explanations are available.We present a unique combination of model- andspectrum-based fault localisation approach to rankexplanations and show that the combined frame-work outperforms the individual approaches as wellas other state of the art automated debugging mech-anisms.

1 IntroductionThe problem of faulty software has been recognised as long-standing issue, with considerable costs attached to locatingand eliminating problems in development as well as afterdeployment of software systems [19]. In particular, testing,validation and debugging of software consumes a considerableslice of the overall software development costs. Hence, numer-ous approaches have been proposed to automate parts of thisprocess to help detect more defects earlier in the developmentcycle and to guide software engineers towards possible faults.

Early debugging efforts were geared towards reducing thesize of a program that must be investigated by analysing thestructure and dependencies between different parts of the pro-gram’s source code [16]. More recently, dynamic analysistechniques have been proposed that exploit traces of programexecutions to accommodate the size of modern software sys-tems [23]. While applicable to a wide variety of programs,both approaches are limited by the absence of a detailed modelof the correct behaviour of a program.

To overcome these limitations, a spectrum of model-basedfault isolation techniques has been advocated as powerfuldebugging aid that can help to isolate faults in complex pro-grams [12]. By comparing the state and behaviour of a pro-gram to what is anticipated by its programmer, model-basedreasoning techniques separate those parts of a program thatmay contain a fault from those that cannot be responsiblefor observed symptoms. A distinguished advantage of themodel-based framework is that it helps programmers by sys-tematically exploring different fault assumptions while hidingthe complex underlying reasoning procedures behind a simpleintuitive conceptual interface.

Initial experiences with model-based software debugging(MBSD) have shown that the approach is competitive withother state of the art automated debugging aids [13], but it hasalso become clear that no single technique is sufficient to dealwith a variety of programs and faults. Instead, a combinationof approaches must be pursued where the strengths of indi-vidual techniques complement each other to lead to a moreaccurate and robust debugging tool.

In this paper we present a combined framework that inte-grates model-based debugging with popular dynamic programanalysis techniques to focus search and rank results. We showthat, as a result, fewer program fragments are being impli-cated, leading to considerably increased accuracy as well asreduced computational complexity of the overall approach.While MBSD is general enough to be combined with almostany debugging tool that can expose its findings in terms ofthe original program’s source code and a set of fault assump-tions, the combination of semantic and trace-based analysis isparticularly appealing, since the approaches contribute com-plementary information: MBSD injects and analyses specificmodifications to the semantics of a program, while dynamicanalysis exploits fault correlation to focus the search.

Our presentation is structured as follows: The principles ofmodel-based debugging are outlined in Section 2, followed bya discussion of spectrum-based fault localisation in Section 3.The combined framework is discussed in Section 4. Empiricalvalidation of our approach and our findings are given in Sec-tion 5. Section 6 discusses relevant related works, followed bythe conclusion.

2 Model-based Software Fault LocalisationIn search for effective (semi-) automated debugging aids,many different strategies have been proposed in the last threedecades. Approaches to automate analysis and isolation offaults in programs range from purely syntactical checks toisolate common fault patterns [4], over execution trace-basedanalysis [15] to full-fledged semi-automatic program verifica-tion [2]. Syntax-based analysis can be easily applied to mostprograms, but its results are often language-specific and de-pend on particular syntactic programming styles; trace-basedtechniques depend on a suitable test harness being available.

Better results can often be achieved if a model of the correctprogram behaviour is available to guide debugging efforts. Forexample, a partial specification expressed in some formal lan-

guage. Unfortunately, building such models is error prone andprohibitively expensive for many software development sce-narios. Attempts to devise formal specifications for non-trivialsystems has shown that constructing a model that captures anabstraction of the semantics of a system can be as difficult anderror-prone as building a concrete implementation [14].

Model-based debugging [12] aims to close the gap be-tween powerful formal analysis techniques and execution-based strategies in a way that does not require the end-user topossess knowledge of the underlying reasoning mechanisms.Here, an adaption of the classic “reasoning from first princi-ples” paradigm borrowed from diagnosis of physical systemsis particularly appealing, since much of the complexity ofthe formal underpinnings of program analysis can be hiddenbehind an interface that resembles the end-user’s traditionalview of software development.1

Different to classical model-based diagnosis, where a cor-rect model is furnished and compared to symptoms exhibitedby an actual faulty physical artifact, debugging software re-verses the roles of model and observations. Instead of relyingon the user to formally specify the desired program behaviour,the (faulty) program is taken as its own model and is com-pared to examples representing correct and incorrect execu-tions. Hence, the model in MBSD reflects the faults present inthe program, while the observations indicate program inputsand correct and incorrect aspects of a program’s execution. Ob-servations can either be introduced interactively, or be sourcedfrom existing test suites.

Example 1 Consider the program in Figure 1. An ob-servation for this program consists of concrete programinputs, that is values for variables tbl, n and k beforeline 1, together with the anticipated result value returnedby the algorithm. For example, the assignments tbl ←[90, 21, 15, 0, 0, 0, 8, 23, 0, 0, 0, 0, 50, 60, 59], n ← 16, k ←90 and the assertion result = 0 could be an “observation”specifying the inputs and the desired result of a particularprogram execution.

Since the result (−1) obtained by running the program onthe given inputs contradicts the anticipated result (0), it hasbeen shown that the program is incorrect. (Indeed, the programcontains a defect in line 9; when assigning 0 to variable i, theprogram works as expected.)

In the following, we briefly outline the model construction.More detailed discussion can be found in [11; 12]. A programis partitioned into “components”, each representing a partic-ular fragment in the program’s source code. The behaviourof each component is automatically derived from the effectsof individual expressions the component comprises. Connec-tions between components are based on control- and data-dependencies between the program fragments represented byeach component.

Example 2 Assume a model at statement granularity is to becreated from the program in Figure 1. For each statements, a separate component is created that is comprised of theexpressions and sub-expressions in s. The inputs and outputs

1In the software engineering context, the notion of “first prin-ciples” may be interpreted in the sense of “directly available fromprogram execution and source code”.

function FINDINDEX(tbl, n, k). Find the index of key k in the hash table

tbl[0, . . . , n− 1], or −1 if not found.Assumes that tbl contains a free slot.

1 i← HASH(k) . Hash key2 while tbl[i] 6= 0 do . Empty slot?3 if tbl[i] = k then4 return i . Found match5 end if6 if i < n− 1 then . At end?7 i← i + 1 . Try next8 else9 i← 1 . Wrap around (incorrectly)

10 end if11 end while12 return −1 . Not found

end function

Figure 1: Algorithm to search in a hash table.(There is a fault in line 9.)

of the components correspond to the used and modified vari-ables, respectively. Connections between the components arecreated to reflect data dependencies between statements inthe program (as determined by a simple data flow analysis).Additional variables and components may be introduced tocorrectly capture data flow at points where control flow pathsmay split or merge.

The component C7 corresponding to statement 7 in Fig-ure 1 is represented as a component with input i2 and outputi7. Here, i7 represents the result value of statement 7, andi2 denotes the previous value of variable i that is implicitlydefined at the loop head in line 2.

Similar to classical model-based diagnosis, the model alsoprovides different operating modes for each component, wherethe “correct” mode ¬AB(C) of component C corresponds tothe case where C is not to blame for a program’s misbehaviour.In this case, C is defined to function as specified in the pro-gram. Conversely, when C is assumed “abnormal” (AB(C)),the C may deviate from the program’s behaviour.

Example 3 The behaviour of C7 can be expressed as the log-ical sentence

¬AB(C7)⇒ i7 = i2 + 1. (1)

In the case where C7 is considered faulty (AB(C7) is true),the effect on i7 is left unspecified.

The main difference between the original program and itsmodel is that the model represents the program in a form thatis suitable for automated consistency checking and predictionof values in program states in the presence of fault assump-tions. This includes program simulation on partially definedprogram states and backward propagation of values or con-straints, which would not occur in a regular (forward) programexecution.

Since the resulting model includes the same faults as theprogram, means to compensate for incorrect structure andbehaviour of components must be introduced. While heuristicsto diagnose structural deficiencies in physical systems can be

based on invariants and spatial proximity [3], in software, themodel must be adapted and restructured once a defect in itsstructure has become a likely explanation. Here, detection andmodel adaption must be guided by using abstract assertionsthat capture simple “structural invariants” [11]. Also, sincedifferent fault assumptions may alter the control and data flowin a program, models may be created on-the-fly rather than inthe initial setup stage.

A trade-off between computational complexity and accu-racy can be achieved by selecting different abstractions andmodels [12], both in terms of model granularity and represen-tation of program states and executed transitions.Example 4 In Example 3 the representation of program statehas been left unspecified. Using an interval abstraction toapproximate a set of values, sentence (1) becomes a constraintover interval-valued variables i2 and i7 [13]. Another possibleabstraction is to encode the operation as logical sentences overthe variables’ bit representations [13]. In this paper, we usethe interval abstraction described in [12], since it providesgood accuracy but avoids the computational complexity of thebit-wise representation.

Similar to consistency-based diagnosis in physical sys-tems [17], from discrepancies between the behaviour predictedby the model and the behaviour anticipated by the user, sets offault assumptions are isolated that render the model consistentwith the observations. Diagnoses are obtained by mappingthe implicated components into the program’s source code.Formally, our framework is based on extensions to Reiter’sconsistency-based framework [12]:Definition 1 (Diagnosis) Let P denote a program and T aset of test cases, where each T ∈ T is a pair 〈I,A〉 where Ispecifies P’s inputs and A is a set of assertions over variablesin P that (partially) specify the correct behaviour of P withrespect to T . Let C denote a partition of the statements in P .

A diagnosis of P with respect to T is a set of componentsD such that ∀〈I,A〉 ∈ T :

P(I)∧{AB(C)|C ∈ D}∧{¬AB(C)|C ∈ C \ D} 6|= ¬A.

Example 5 Continuing Example 4, a contradiction betweenthe test case from Example 1 and the program is detected whenthe assertion checking the expected result fails. It is derivedthat the (cardinality-)minimal fault assumptions that are con-sistent with our test specification are: {AB(C1)}, {AB(C7)},{AB(C9)}, and {AB(C12)}. Hence, the statements in lines 1,7, 9 and 12 are considered the possible root causes of the symp-toms. Any other statement cannot alone explain the incorrectresult, since the result remains incorrect even if a statement isaltered.

2.1 Issues in MBSDWhile the pure MBSD framework is well-suited to carry outcomplex inferences, its application in practice is limited dueto the following factors:

Result interpretation: If many explanations are returned,MBSD alone provides little information to discriminate be-tween the different explanations. Here, a mechanism to rankresults would be desired.

Different to electronic circuits, where long sequences of e.g.inverters are uncommon, program executions frequently con-tain long chains of control- and data dependencies, leading toa number of explanations that cannot be distinguished withoutfurther observations. For example, the value of the conditionaltest in line 2 depends on all statements executed in previousiterations. Interactive measurement selection techniques aredifficult to apply, since program states in different executionsmay be incomparable, rendering entropy-based solutions inef-fective.Returning a “super component” as explanation is alsonot viable in general, since the involved statements can spanmany different program fragments. Therefore, an approachthat works with little or no user involvement is desired.

Scalability: The application of MBSD has been limited tosmall programs, since the computational effort exceeds whatis considered reasonable for interactive scenarios. Hence,inference processes must be applied selectively to remainefficient.

External interfaces: MBSD requires that effects of pro-gram fragments can be simulated even if only partial informa-tion is available. Programs interacting with external compo-nents, such as I/O, files and GUIs, must be modified to eitherremove these interactions or provide placeholder implementa-tions.

The first two issues can be addressed by introducing a mech-anism to estimate, for each component C in the model, howlikely it is that C contains a fault. The third issue is commonto most program analysis techniques and is beyond the scopeof this paper.

Assuming a suitable measure is available, ranking of resultsbased on fault probability and investigating different explana-tions in best-first order rather than computing all explanationsat once are straight forward. Since a priori probabilities aretypically not directly available, other means to determine asuitable likelihood value must be used.

The aim of this paper is to show that correlation betweenthe execution patterns of statements with correct and failedexecutions can significantly improve diagnosis results. Thefollowing section outlines our approach to assessing the simi-larity between different program executions and test outcomes.Since MBSD does not usually exploit correct program ex-ecutions in any way, this approach can contribute valuableinformation to guide the model-based framework.

3 Spectrum-based Fault LocalisationWhen more than one test case are available, dynamic programanalysis techniques have shown that comparing the programbehaviour over multiple test runs can indicate which programcomponents may be likely to contribute to an observed symp-tom.

In the following, we assume that a program P comprises aset of M = |C| components (statements in the context of thispaper) and is executed using N = |T | test cases that eitherpass of fail.

Program (component) activity is recorded in terms of pro-gram spectra [1; 8]. This data is collected at run-time, and

M components errorvector

N spectra

o11 o12 . . . o1M e1

o21 o22 . . . o2M e2

......

. . ....

...oN1 oN2 . . . oNM eN

Figure 2: Participation Matrix O

typically consists of a number of counters or flags for thedifferent components of a program. We use the so-called hitspectra that indicate whether a component was involved in a(test) run or not.

Both spectra and pass/fail information is input to a spectrum-based fault localisation (SFL) mechanism. The combinedinformation is expressed in terms of the N × (M + 1) par-ticipation matrix O (see Figure 2). An element oij is equalto 1 if component j took part in the execution of test run i,and 0 otherwise. The rightmost column of O, the error vectore, represents the test outcome. The element ei = oi,m+1 isequal to 1 if run i failed, and 0 if run i passed. For j ≤M , therow Oi∗ indicates whether a component was executed in run i,whereas the column O∗j indicates in which runs component jwas involved.

In SFL one measures the similarity between the error vec-tor e and the activity profile vector O∗j for each compo-nent j. This similarity is quantified by a similarity coeffi-cient, expressed in terms of four counters apq(j) that countthe number of positions in which O∗j and e contain respec-tive values p and q; that is, for p, q ∈ {0, 1}, we defineapq(j) = |{i | oij = p ∧ ei = q}|. In this paper, the Ochiaisimilarity coefficient is used, known from molecular biology,since previous investigations have identified it as the best co-efficient to be used for SFL [1]. It is defined as

sj =a11(j)√

(a11(j) + a01(j)) ∗ (a11(j) + a10(j))The similarity coefficient sj associated with each componentCj indicates the correlation between the executions of Cj andthe observed incorrect program behaviour. Applying the hy-pothesis that closely correlated components are more likelyto be relevant to an observed misbehaviour, sj can be reinter-preted as “fault probability” and components can be ranked.Example 6 Executing the program in Figure 1 using the testcase described in Example 1, the first row vector in the partici-pation matrix below is obtained: The vector contains a single0 entry, indicating that all components but C4 are executed.Since the returned value does not match the anticipated result,the entry in the error vector is set to 1.

Assume that further tests are executed to yield the followingparticipation matrix:

C1 C2 C3 C4 C6 C7 C9 C12 e26666641 1 1 0 1 1 1 11 1 0 0 0 0 0 11 1 1 1 0 0 0 01 1 1 1 1 1 1 01 1 1 0 1 1 1 11 1 1 0 1 1 1 0

˛˛˛

100001

37777750.58 0.58 0.63 0.00 0.71 0.71 0.71 0.58

For each component the Ochiai similarity sj is given below thematrix. For C3, the similarity coefficient s3 is 0.63: as can be seenfrom the third column in the matrix, there are two failing test runswhere C3 is executed (a11(3) = 2), no failing run where C3 doesnot participate (a01(3) = 0), and three successful executions whereC3 is involved (a10(3) = 3).

C6, C7 and C9 are considered to be most closely correlated withfailing tests and should be examined first. Conversely, C4 is notconsidered relevant at all.

Recent studies on spectra-based fault localisation indicatedthat this scheme is effective even for small test suites contain-ing only few test cases [1]. For the programs investigated here,good fault localisation was achieved when using six failingtest cases and twenty passing runs.

4 Spectra-Enhanced MBSDAs SFL functions without a semantic model of the program,the technique is easily applied. Experiments with differentsimilarity measures have shown that Ochiai similarity gener-ally outperforms other spectra-based indicators and can givegood hints on the location of a fault in a program [1].

At the same time, the absence of a model also limits theaccuracy of fault localisation. Even for comprehensive testsuites, the execution patterns of some components may notbe distinguishable, and faulty components may show erro-neous behaviour only in particular execution contexts. As aresult, the similarity measure may implicate unrelated programfragments.

Conversely, the model-based technique captures the seman-tics of programming constructs, but does not assign rankinginformation to candidate explanations. Furthermore, model-based diagnosis traditionally only considers discrepancies, butdoes not utilise correct test cases, although all failing test caseswill be considered. In contrast, spectra-based methods exploitboth correct and failing test runs to rank candidates. Hence,both techniques complement each other.

Algorithm 1 outlines our combined approach. The algo-rithm executes in three stages, with the similarity-based ap-proach used in the setup stage (steps 1 to 5), feeding intothe subsequent model-based filtering stage (steps 6 to 16),followed by an optional best-first search stage (lines 17 to24). This combination has significantly lower resource re-quirements than applying MBSD on the whole program butusing SFL only to rank results. We start by partitioning theprogram P into a set of components C and execute P on theavailable test cases T to obtain the participation matrixM.UsingM, we partition T into passing tests (TP ) and failingones (TF ). FromM, the Ochiai similarity vector is computed;its values are subsequently assigned to components as a-priorifault probabilities to yield the component list C sorted by faultprobability.2

In the subsequent loop, candidate explanations are com-puted using the MBSD approach to isolate the most likelyexplanations based on C and TF . While it is possible to ap-ply MBSD once to compute all explanations and present theranked candidates to the user, an incremental strategy permits

2We use the term probability as synonym for likelihood to beincorrect. Our measure does not necessarily conform to the laws ofprobability theory.

Algorithm 1 Spectra-Enhanced MBSD AlgorithmInputs: Program P , set of test cases TOutput: Fault assumptions explaining failed test runs

1 C ← CREATECOMPONENTS(P )2 M← GETCOMPONENTMATRIX(C, T )3 〈TP , TF 〉 ← T4 S ← COMPUTESIMILARITY(M)5 C ← ASSIGNCOMPONENTPROBABILITIES(C, S)6 t← 1 + ε7 R← C8 repeat9 D ← MBSD(C, TF , t)

10 if Dbug ∈ D is confirmed then11 return Dbug

12 else13 t← PROBABILITY(D) for some D ∈ D14 end if15 R← R \

⋃D∈D

COMPS(D)

16 until D = ∅ or t = 017 whileR 6= ∅ do18 C ← arg max

C∈R∧∃N∈NEIGHBOURS(C):N /∈RPROBABILITY(C)

19 if C is confirmed faulty then20 return {AB(C)} . Partial explanation21 end if22 R← R∪ (NEIGHBOURS(C) ∩R) \ {C}23 end while24 return No explanation found

the algorithm to stop early once a fault has been identified.In each iteration, the user is presented a number of candidateexplanations for examination. If the actual fault has been lo-cated, the algorithm stops; otherwise, none of the candidatesrepresent valid explanations and other candidates must be gen-erated. The algorithm stops once no more explanations couldbe found or if none of the remaining components was executedfor a failing test.

We modified the basic MBSD algorithm to return only theexplanations with probability p less than a given thresholdt, with the additional restriction that p is maximal amongthe returned diagnoses. Hence, only explanations with thesame likelihood are returned in an iteration of Algorithm 1.Initially, t is set to a value slightly larger than 1 (the Ochiaisimilarity is always ≤ 1), hence candidate explanations withmaximum likelihood are enumerated first. By decreasing t ineach iteration, lower-scoring alternatives are explored if nohigher-scoring candidate has been confirmed by the user. Ourimplementation of MBSD caches intermediate models andconflicts to avoid repeated computations.

If no explanation is found after all components implicatedby MBSD have been explored, we employ a best-first searchprocedure that traverses the program along dependencies be-tween components with decreasing fault probability. No ex-planation may be found if a fault in the program has largercardinality than the MBSD threshold or if the fault affectscomponent inter-dependencies such that the fault assumptions

and model abstraction can no longer represent the fault. Func-tion NEIGHBOURS(C) returns the set of components that aredirectly connected to C in P , COMPS(D) returns the set ofcomponents that occur in diagnosis D, and PROBABILITY(C)returns the fault probability assigned to component C by SFL.In line 18, the component with maximum fault probability thatis connected to a previously explored component is selected.If the component is confirmed to be (part of) a valid expla-nation, the search stops and the diagnosis is returned. Notethat the explanation may only cover part of the true fault. Line24 in Algorithm 1 can only be reached if the faulty programfragment is not covered by any component, or if the user ora-cle that decides whether an explanation is indeed a satisficingexplanation is imperfect and may miss a fault.

We employ the common assumptions that components mayfail independently. While faults in a statement can imply fail-ure in subsequent statements due to data dependencies, thisneed not be true in general; since most faults in our test suiteare confined to a single component and only few statements oc-cur where implied faults are possible, this assumption has notsignificantly affected the outcomes of our study. Also, sincefault probabilities are estimated from correlation with failingtests, different components participating in the same failurewill be assigned higher similarity, partially compensating formissed component fault interactions.

Example 7 Applying Algorithm 1 using the test suite fromExample 6, {AB(C7)} and {AB(C9)} form the set of can-didate explanations. Both candidates are associated with thehighest similarity coefficient 0.71.

Notably, this result improves upon both individual faultlocalisation procedures. Different from pure SFL, {AB(C6)}is no longer considered an explanation. Conversely, candidates{AB(C1)} and {AB(C12)} obtained using pure MBSD arelow-ranking in SFL and hence omitted at this stage. (AB(C12)is already eliminated by pure MBSD when using the secondfailing test case introduced in Example 6.)

Without further information, neither approach can discrimi-nate between the two remaining candidate explanations. Sinceit is assumed that the user acts as oracle that can reliably recog-nise true faults, the algorithm stops after the first iteration,once the statement in Figure 1 corresponding to {AB(C9)}has been confirmed to be incorrect.

Otherwise, the diagnosis threshold t would be set to 0.71and the algorithm would continue to present {AB(C1)} as the(last) remaining alternative explanation.

The use of similarity measures to guide diagnosis can po-tentially lead to considerable savings; moreover, the behaviourof Algorithm 1 degrades gracefully if components with highprobability do not actually correspond to faults. In the worstcase, the number of diagnoses to be examined by the user isthe same as when using the non-guided MBSD strategy. In thenext section we evaluate our algorithm on a larger test suite.

5 Empirical EvaluationTo gain a better understanding of the combined approach, theTCAS program was taken from the Siemens Test Suite3, a test

3http://www-static.cc.gatech.edu/....../aristotle/Tools/subjects/

bench commonly used in the debugging community. Theprogram simulates the resolution-advisory component of a col-lision avoidance system similar to those found in commercialaircraft. The program consists of 138 lines of C code and takestwelve parameters as input; the numeric result value encodesone out of three possible resolution advisories. The programis equipped with 1608 test cases and 41 different variants withknown faults. For each variant, on average, forty test casesreveal a fault. In our experiments, all available test cases wereused.

As the efficiency of debugging in practice often depends onthe experience of the software developer, comparing differentapproaches is difficult. In particular, simple precision andrecall-based evaluation may be insufficient, since the structureof a program is not taken into account. In an attempt to devisean objective measure to assess the quality of automated debug-ging aids, a quality metric that exploits dependencies within aprogram has been proposed, where the quality of a report de-pends on the fraction of a program that need not be examinedgiven the debugging tool’s output. Starting with the programelements implicated by an automated debugger, dependen-cies between program elements are traversed in breadth-firstorder until the fault has been reached. This strategy aimsto mimic programmer behaviour, where possible influencesalong control- and data flow paths are explored. The fractionof the program that has not been traversed leads to the qualityindicator [18].

In our framework, if the explanation covering the true faultis ranked nth among the candidate explanations, the fractionof the program that is traversed is given by

q =∑n

i=1 |STMNTS(Di)||P|

,

where Di represents the ith-ranked candidate explanation,STMNTS is a function that returns the set of program state-ments covered by the components in a diagnosis, and |P|denotes the number of statements in the program. Otherwise,if Algorithm 1 stops without locating the true fault, the breadth-first search procedure is invoked starting with all implicatedstatements ⋃

D∈MBSD( bC,TF ,1+ε)

STMNTS(D)

to obtain q as outlined previously.

5.1 Experimental ResultsResults for the individual approaches have already been pub-lished elsewhere; an evaluation of MBSD is presented in [13],where an interval abstraction is applied to yield, on average,nine statements to be inspected. The median quality indicatoris 0.87. The results obtained with SLF are discussed in [1].Exploring the program beginning with the highest-ranked state-ment, the true fault is located after 28 statements have beenexamined on average. The resulting median report quality is0.86. Figures 3 and 4 summarise the overall outcomes.

To obtain a first impression how well the two approachescomplement each other, Figure 3 contrasts the componentsimplicated by either model with those blamed by both. Itcan be seen that the AIM significantly reduces the numberof components, and that neither model subsumes the other.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

010

2030

4050

60

both MBSD only SFL only

Figure 3: Components implicated by SFL and MBSD

Restricting the debugging process to those statements that ap-pear in both models, the median number of statements reducesfrom 36 to 8. Hence, the total number of statements consid-ered relevant reduces considerably when using the combinedapproach.

Similar improvements can be observed in the ranking ofcomponents. Using the pure SFL approach one hits the truefault after inspecting twenty statements on average, but manyunrelated statements must be examined. When using MBSDas filtering mechanism, the true fault is located after sevenstatements on average. Hence, the model-based filtering mech-anism seems well-suited to prune away irrelevant componentsfrom the SFL fault profiles.

The improved accuracy of the combined approach also re-flects in much improved quality indicators. Figure 4 depictsthe quality measure obtained for the individual 41 test pro-grams using our fault localisation approaches. It is observedthat the combined approach largely outperforms the individualtechniques. In some cases, SFL outperforms the combinedapproach, suggesting that the model used in MBSD may notbe able to accurately reflect the fault. This difference mayalso be attributed to the execution of some faulty code cor-relating well with the failing test cases. If the MBSD partof our algorithm cannot precisely locate the fault, the SFLmethod can score higher. So far, we have not been able todevise heuristics that can consistently predict this discrepancyfrom the a-priori component probabilities and diagnoses tofurther improve accuracy. Since both approaches use heuris-tics to rank candidates, it may be the case that one methodoutperforms the combination on individual candidates, eventhough it is superior overall.

Overall, the fraction of the program that must be inspectedreduces from 13% and 30%, respectively, to 8%. AlthoughMBSD alone is not able to locate faults for 9 of the 41 pro-grams (due to limitations on faults in global variable initialisa-tion in our current implementation), the overall performance ofthe combined approach does not seem to be adversely affectedin most cases. This can be explained by two observations: (i)the number of diagnoses that are implicated in those cases issmall (4 on average), and (ii) the suspect program fragmentsare close to the actual faults when navigating the programstructure.

q va

lue

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

TCAS Variants

SFLMBSDSFL+MBSDTop6

Figure 4: Report quality

We also evaluated a modified version of Algorithm 1, wherethe MBSD section is stopped after the six4 most highly rankedcomponents have been explored; the remaining componentswere subsequently explored using the best-first part of ouralgorithm. The resulting quality indicators are labelled Top6 inFigure 4. The results indicate that the components implicatedby the combined approach sometimes narrowly miss the truefaults; in these cases, the score measure improves compared tothe combined approach. In other cases, following the originalalgorithm is more successful. Overall, the quality indicators donot differ significantly between the two models. Investigatingwhether heuristics can be developed that choose a cutoff toimprove accuracy remains for future work.

Figure 5 visualises the number of located bugs for differentfractions of inspected code. Our approach vastly outperformsthe simple spectrum-based fault localisation techniques pro-posed in [18], where different combinations of union and inter-section of “similar” passing and failing test runs are computed.This can be attributed to the improved ranking mechanismsbuilt into our algorithm that is more robust with respect tooverlapping passing and failing executions. Our combined ap-proach also improves with respect to SOBER [10], a statisticalapproach based on hypothesis testing that has been shown todominate other recent bug detectors.

Delta slicing and explain [7] are two techniques for fault lo-calisation that exploit differences between passing and failingabstract program executions traces found by a model checker.Comparing our results to the published results in [7], we con-clude that the combination of SFL and MBSD is far superiorthan explain (which requires the user to explore 24–64% of aprogram) and performs competitive with respect to Delta Slic-ing (within 5%). Interestingly, when using the cutoff variant ofor our algorithm described in Figure 4, our approach also dom-inates Delta Slicing. (This comparison is not exhaustive, sinceresults for only a small subset of the examples considered inour study is available for the competing approaches.)

4This cutoff seemed to have the best overall effect for an extendedtest suite used in [1].

Percentage of inspected code

Num

ber

of lo

cate

d bu

gs

05

1015

2025

30

[0,10)[10,20)

[20,30)[30,40)

[40,50)[50,60)

[60,70)[70,80)

[80,90)[90,100]

NNSoberSFL+MBSD

Figure 5: Debugging efficiency

6 Related WorkSeveral systems employing dynamic analysis techniques forfault localisation are present in the literature. Tarantula [8] ob-tains program spectra from test case executions and graphicallyvisualises the fault proneness indicator based on participationof individual statements in passing and failing runs. Tarantuladoes not exploit information about the anticipated behaviourof a program and hence relies on external tools to assess theoutcome of test runs.

Machine learning techniques have been applied to pro-grams [21] and their executions [15] to infer likely invariantsthat must hold at particular locations in a program. Violationscan subsequently be used to detect potential errors. Model-based approaches have been shown to provide more reliablebehaviour than [15], since success of the trace analysis de-pends much on the test runs and type of invariants to be in-ferred [9]. The static program analysis approach requires thatsimilar patterns appear repeatedly in a program, but is notapplicable when common patterns are not easily identified.

Combining program execution and symbolic evaluationhas been proposed to infer possible errors [5]. Similar toMBSD, a symbolic, under-constrained representation of aprogram execution and memory structures are built. Insteadof using fault probabilities to guide diagnosis, only thosecandidate explanations that definitively imply a test failureare flagged. Hence, the tool complements our approach byhighlighting a subset of all provable faults in a program, whileour approach aims at identifying those program fragments thatmay contribute to a fault.

Model-based debugging has been explored using a varietyof different abstractions of concrete programs [12]. Recently,similar techniques have also been proposed to isolate specificfaults stemming from incorrect implementation of high-levelconceptual models [22]. Mutations applied to state machinemodels allow to detect conceptual errors, such as incorrectcontrol flow and missing or additional features found in theimplementation compared to its specification. Model-basedtest generation [6] from abstract specifications of systems

employs a similar idea where possible faults manifested asdifferences in abstract state machines are analysed to generatetests. Our work differs in that we are concerned with programrepresentations that more closely reflect the actual programartefact to locate faults at a more detailed level. While initialsteps to integrate similar conceptual abstract models have beenundertaken in an attempt to isolate “structural” faults [11],detailed analysis remains future work.

Diagnosis and repair in the context of distributed systemscomposed from Web Services has also been investigated [20].In particular, diagnosability and analysis of diagnosis andrepair plans are central parts of this work. Similar ideas are inprinciple applicable in the debugging context, but further workis required to devise a suitable analysis framework that canoperate on the more implementation-centric view employedin this work.

7 Conclusions & Future WorkWe have shown that the accuracy of model-based debuggingincreases significantly when applied in combination with com-plementary approaches that estimate fault probabilities. Theunique combination of semantics-based analysis as undertakenin MBSD and dynamic aspects obtained from program exe-cution spectra has proved to focus debugging efforts; overall,a reduction of user effort to less than 10% compared to thecomplete program has been achieved on our test suite. Wehave further shown that our approach is among the state of theart automated debugging tools.

Several issues for further research remain: On the MBSDside, connecting the lower-level models that reflect the pro-gram to high-level conceptual models to detect a more diverseset of faults seems promising to address current limitations.The same idea may be useful to aid model selection and focusfault assumptions. On the dynamic analysis side, introducingmachine learning techniques to infer likely invariants that canthen be used to further filter and guide the MBSD modellingefforts are possible avenues worth further exploration.

References[1] Rui Abreu, Peter Zoeteweij, and Arjan J. C. van Gemund.

On the accuracy of spectrum-based fault localization. InTAIC PART’07, pages 89–98. IEEE, 2007.

[2] Thomas Ball and Sriram K. Rajamani. The SLAMproject: debugging system software via static analysis.In POPL, pages 1–3, 2002.

[3] Claudia Bottcher. No faults in structure? How to di-agnose hidden interaction. In Proc. 14th IJCAI, pages1728–1735, 1995.

[4] Ian F. Darwin. Checking C programs with lint. O’Reilly& Associates, Inc., 1986.

[5] Dawson R. Engler and Daniel Dunbar. Under-constrained execution: making automatic code destruc-tion easy and scalable. In Proc. ACM ISSTA, pages 1–4.ACM, 2007.

[6] Michael Esser and Peter Struss. Automated test genera-tion from models based on functional software specifica-tions. In Proc. 20th IJCAI, pages 2255–2268, 2007.

[7] Alex Groce, Sagar Chaki, Daniel Kroening, and OferStrichman. Error explanation with distance metrics.STTT, 8(3):229–247, 2006.

[8] James A. Jones and Mary Jean Harrold. Empirical eval-uation of the tarantula automatic fault-localization tech-nique. In ASE, pages 273–282. ACM Press, 2005.

[9] Daniel Kob and Franz Wotawa. A comparison of faultexplanation and localization. In Proc. DX’05 Workshop,pages 157–162, 2005.

[10] Chao Liu, Xifeng Yan, Long Fei, Jiawei Han, andSamuel P. Midkiff. SOBER: statistical model-based buglocalization. In ESEC/SIGSOFT FSE, pages 286–295.ACM, 2005.

[11] Wolfgang Mayer. Static and Hybrid Analysis in Model-based Debugging. PhD thesis, School of Computerand Information Science, University of South Australia,2007.

[12] Wolfgang Mayer and Markus Stumptner. Model-baseddebugging - state of the art and future challenges.ENTCS, 171:61–82, 2007.

[13] Wolfgang Mayer and Markus Stumptner. Models andtradeoffs in model-based debugging. In Proc. DX’07,2007.

[14] Madanlal Musuvathi and Dawson R. Engler. Somelessons from using static analysis and software modelchecking for bug finding. ENTCS, 89(3), 2003.

[15] Jeremy W. Nimmer and Michael D. Ernst. Static verifi-cation of dynamically detected program invariants: Inte-grating Daikon and ESC/Java. In Proc. RV’01, 2001.

[16] Hsin Pan and Eugene H. Spafford. Heuristics for auto-matic localization of software faults. Technical report,Purdue University, 1992.

[17] Raymond Reiter. A theory of diagnosis from first princi-ples. Artificial Intelligence, 32(1):57–95, 1987.

[18] Manos Renieris and Steven P. Reiss. Fault localizationwith nearest neighbor queries. In ASE’03, pages 30–39.IEEE, 2003.

[19] RTI. Planning report 02-3: The economic impacts ofinadequate infrastructure for software testing. PlanningReport 02-3, NIST, 2002.

[20] WSDIAMOND. WS-Diamond deliverable D5.1: Char-acterization of diagnosability and repairability for self-healing web services. Technical Report IST-516933,University of Torino and others, 2007.

[21] Yichen Xie and Dawson R. Engler. Using redundanciesto find errors. IEEE TSE, 29(10):915–928, 2003.

[22] Cemal Yilmaz and Clay Williams. An automated model-based debugging approach. In ASE, pages 174–183.ACM Press, 2007.

[23] Peter Zoeteweij, Rui Abreu, Rob Golsteijn, and A. J. C.van Gemund. Diagnosis of embedded software usingprogram spectra. In Proc. ECBS’07. IEEE, 2007.