1

Pathological Motion Detection for Robust MissingData Treatment

David Corrigan, Naomi Harte, Anil Kokaram

Abstract— This paper presents a new missing data detectionalgorithm that is robust to Pathological Motion (PM). PM causesclean image data to be misdiagnosed as missing data, resulting indamage to the image during restoration. The proposed algorithmuses a probabilistic framework to jointly detect PM and missingdata. It builds on an existing technique of using five framesfor detection instead of the standard three frames. This allowsthe temporally impulsive intensity profile of blotches to bedistinguished from the quasi-periodic profile of PM. Anotherdiagnostic for PM is defined on the motion fields of the fiveframe window. This follows the observation that PM results inmotion fields which are not smooth. A ground truth comparisonwith standard missing data detectors shows that the proposedalgorithm dramatically reduces the number of falsely detectedmissing data regions. It is also shown to reduce image damageduring missing data treatment.

I. I NTRODUCTION

Missing data treatment is an important process in the digitalrestoration of archived media. It is concerned with removalofvisible impairments, known as blotches, commonly present inarchived sequences. Also referred to as “dirt and sparkle”,blotches are caused by either adherence of dirt to the surfaceof the film emulsion, or by abrasion of the emulsion. Con-sequently, blotches manifest as typically small dark or brightregions in the digital sequence which obscures the true imagedata. Missing Data Treatment (MDT) algorithms attempt todetect these blotches and to somehow recover the true imagedata.

Most blotch detection algorithms are derived from theassumption that if a blotch is present at a particular locationin a frame, then no blotches are present at the same point inneighbouring frames. Blotches are therefore considered tobe atemporally impulsive event due to the change in intensity fromthe true intensity to the blotch intensity at a particular pixel.Detection of blotches can be performed by searching for siteswhere temporal discontinuities exist between the current frameand the previous and next frames. Consequently, missing datadetectors employ a three frame window and detect disconti-nuities by measuring the Displaced Frame Difference (DFD)between the appropriate frames. Examples of detectors using athree frame window include the deterministic Spike DetectionIndex (SDI) [1], [2] and Rank Order Distance (ROD) [3], [4]detectors, which apply deterministic thresholds to the DFDs to

David Corrigan is with the Department of Electronic and Electrical Engi-neering, Trinity College Dublin, Dublin 2, Ireland (email:corrigad@tcd.ie).

Naomi Harte is with the Department of Electronic and Electrical Engineer-ing, Trinity College Dublin, Dublin 2, Ireland (email: nharte@tcd.ie).

Anil Kokaram is with the Department of Electronic and Electrical Engineer-ing, Trinity College Dublin, Dublin 2, Ireland (email: anil.kokaram@tcd.ie).

Fig. 1. Left: Three consecutive frames of a sequence where the motion of thepropellor blades is PM (repetitive occlusion); Right: The images after MDTusing [8]. The arrows indicate the removal of the blades due to the bladesbeing misdiagnosed a blotch.

detect discontinuities, and the probabilistic Morris [5],[2] andBlotch MRF [6] algorithms, which allow for the inclusion ofprior knowledge of the result into the decision. Other detectors[1], [7] propose methods for the estimation of a non-binaryblotch confidence index.

Motion estimation is a necessary part of the blotch detec-tion process. It is used to compensate for motion over thethree frame window, preventing moving objects from beingconfused with blotches. Although the state of the art motionestimators can robustly estimate the most common motionpatterns, there are complex motion patterns which can notbe estimated accurately. This complex motion is referred toasPathological Motion(PM) and can lead to true image databeing detected as a blotch (See Fig. 1). Pathological Motionis usually caused by the occlusion and uncovering of objectsin a sequence, or by fast moving objects (resulting in thephenomenon of motion blur). More extensive taxonomies ofPM are presented in [9], [10]. PM affects the robustness ofblotch detectors as it can also cause temporal discontinuities.

2

Fig. 2. This figure illustrates the arrangement of the five frames window forblotch detection (in blue). The window is centred on framen (In) which is theframe on which blotch detection is to be performed. Between the five frames,four temporal discontinuity fields (t∗,∗) (in red) can be estimated betweenpairs of consecutive frames. In this diagram a blotch is present on framen andthe yellow object is occluded in every second frame (an example of PM). Dueto the pattern of occlusion it is impossible to distinguish between the blotchand the object when detecting blotches based on temporal discontinuities overa three frame window. By extending the window it is possible to perform acorrect diagnosis.

In this paper, a new missing data detection algorithm ispresented. The algorithm is made more robust by preventingPM regions being detected as blotches, thereby preventingdamage to image data during restoration of dirty sequences.A five frame window is used instead of the standard threeframes. This allows blotches to be distinguished from longterm forms of PM, since blotches have a temporally impulsiveintensity profile and PM often causes the intensity to vary ina periodic manner (See Fig. 2). The next section introduces areview of the existing approaches to improving the robustnessof blotch detection to PM and outlines the motivation for thenew algorithm. The algorithm is then outlined in Sections IIIand IV. Section V presents an evaluation of the algorithm,including the results of an experimental comparison of thenew detector with existing blotch detectors on two groundtruth sequences. The paper concludes with a discussion of thealgorithm, outlining areas for future development.

II. REVIEW OF PM DETECTION ALGORITHMS

Previous approaches for dealing with PM have largely falleninto three categories. The first strategy is to recognise that PMis associated with the motion of foreground objects. Therefore,by performing a foreground/background segmentation, likelyPM regions can be associated with the estimated foregroundand missing data detection can be made more conservativein these regions [6]. The second uses the colour statisticsof blotches and PM regions (specifically motion blur) todistinguish between them [11]. This is possible as blotchestypically manifest as regions of dark or bright intensity whilemotion blur is the apparent smearing of a foreground objectover a background.

The final approach is to extend the temporal window forblotch detection from three to five frames. This approach wasfirst adopted by van Roosmalen in [12], in which the RODdetector was modified to operate on a five frame window.Another more elaborate algorithm adopting this approach is

outlined by Bornard in [10]. In the Bornard algorithm aprobabilistic framework is used to estimate the four binarytemporal discontinuity fields over a five frame window (SeeFig. 2). This is followed by a deterministic interpretationstep to perform the classification. Missing data is classified iftemporal discontinuities only exist at a site in the two centraldiscontinuity fields. Sites at which discontinuities also exist inthe two outer fields are classified as PM.

The algorithm proposed in this paper adopts a similarstrategy to the Bornard algorithm with two key modifications.Instead of having separate temporal discontinuity detectionand interpretation stages, these stages are integrated into asingle probabilistic framework which estimates a blotch andPM mask from the five frames. This allows for prior knowl-edge of the blotch and PM masks as well as the TemporalDiscontinuity Fields (TDFs) to be included in the framework.The second novelty to algorithm is the addition of a furtherdecision criterion for PM based on the smoothness of themotion fields. This follows from the observation that PMcauses the local smoothness assumption of a motion field to beviolated. The vector field divergence is used as the smoothnessmeasure in the proposed algorithm.

The following section outlines the blotch and PM detectionmodel employed in the algorithm and how this relates to thetemporal discontinuity and motion field smoothness informa-tion. Section IV describes how the two detection criteria areintegrated into the probabilistic framework and subsequentlyintroduces the new algorithm.

III. A LGORITHM OVERVIEW

The goal of the algorithm is to segment the current frameinto regions of PM, missing data and uncorrupted sites. Toachieve this, a label field,l(x), is defined as follows

l(x) =

0 No missing data or Pathological Motion

1 Missing Data

2 Pathological Motion.

(1)

Every pixel in the image is assigned one of the three labels.Like the Bornard algorithm, a pixel can not be both a missingdata site and PM site. Although there is no reason why ablotch could not occur in a PM region, it would not bepossible to reliably detect a blotch using a purely temporaldiscontinuity based detector. Furthermore, a PM detectionimplies that the data at any blotch cannot be interpolatedtemporally. Consequently, interpolation must be performedusing spatial rather than temporal information.

A. Temporal Discontinuity Based Detection

Like the Bornard algorithm, the DFD between a pair ofneighbouring frames is used as the measure of temporaldiscontinuity. A temporal discontinuity is said to exist atagiven site if the absolute value of the DFD at the site issufficiently large. In the five frame window, there are fourpairs of neighbouring frames where the frames are denotedby In−2(x), In−1(x), In(x), In+1(x) and In+2(x). A DFDcan be measured between each pair, with a binary Temporal

3

TABLE I

THE TEMPORAL DISCONTINUITY MODEL FOR THE FRAMEWORK.

State, s(x) t(x) l(x) State, s(x) t(x) l(x)0 0, 0, 0, 0 0 8 1, 0, 0, 0 01 0, 0, 0, 1 0 9 1, 0, 0, 1 22 0, 0, 1, 0 0 10 1, 0, 1, 0 23 0,0,1,1 2 11 1, 0, 1, 1 24 0, 1, 0, 0 0 12 1,1,0,0 25 0, 1, 0, 1 2 13 1, 1, 0, 1 26 0,1,1,0 1 14 1, 1, 1, 0 27 0, 1, 1, 1 2 15 1, 1, 1, 1 2

Discontinuity Field associated with each DFD. Consequently,four DFDs (∆n−2(x), ∆n−1(x), ∆n+1(x) and ∆n+2(x))and TDFs (t(x) = [tn−2(x), tn−1(x), tn+1(x), tn+2(x)])are estimated over the five-frame window. In the proposedalgorithm, all configurations of temporal discontinuity areconsidered. There are sixteen possible configurations of thefour TDFs and a state fields(x) is defined which describes theconfiguration of a site. Each configuration is directly mappedto a value ofl(x) (Table I).

Missing data regions are characterised by an impulsivetemporal intensity profile. Therefore, if a blotch exists inframen, the absolute values of the DFDs between both framesn andn− 1 (∆n−1(x)) and framesn andn + 1 (∆n+1(x)) will belarge but the other two DFDs will have small values (SeeFig. 2). This results in a predictable temporal discontinuityprofile described bys(x) = 6 in table I. On the otherhand, the quasi-periodic profile of long-term PM will resultinhigh absolute values in more than two DFDs. Consequently,any temporal discontinuity configuration with two or moretemporal discontinuities present, apart from the missing datacase, is considered to be associated with PM (Table I). In thetable, states 3 and 12 can correspond to missing data in theneighbouring frames. It is considered here as PM as it is notcaused by missing data in the current frame.

In the Bornard algorithm, all candidate blotch sites arefound (i.e. wheretn−1(x) = tn+1(x) = 1) before eachcandidate is classified as either being a blotch site or PM site.This is performed based on the frequency of discontinuity intn−2(x) and tn+2(x) in the neighbourhood of the candidate.Effectively, this means that not every configuration of temporaldiscontinuity is detected, distinguishing the Bornard algorithmfrom the proposed algorithm.

B. Motion Field Smoothness Based Detection

The second feature used to detect PM is the smoothnessof the motion field. The vector field divergence is usedas the basis of the smoothness measure. By estimating thedivergences of all the motion fields in the five-frame window,a smoothness energy term,Ediv(x), can be defined whichdescribes the smoothness of the motion fields. This measure ofdivergence will be large in the presence of long-term PM andlow in other cases. The divergence measure is associated witha binary fieldv(x), where a value of0 corresponds to a lowdivergence measure (i.e. when motion is not pathological) anda value of1 corresponds to a high divergence measure (i.e.for PM). This relationship is described in Table II.

TABLE II

THE DIVERGENCE STATE MODEL FOR THE PROBABILISTIC FRAMEWORK.

Divergence State,v(x) l(x) s(x)0 0,1 0, 1, 2, 4, 6, 81 2 3, 5, 7, 9, 10, 11, 12, 13, 14, 15

The fieldss and v act as auxiliary variables which dictatethe final value ofl(x) in the probabilistic framework. For agiven value ofs or v, the value ofl is directly defined throughtables I and II.

IV. PROBABILISTIC FRAMEWORK

The framework derives an estimate ofs(x) from the pos-terior P (s(x)|∆n(x), Ediv(x)). For convenience of notation,the four DFDs have been grouped into a vector valued func-tion ∆n where ∆n(x) =

[

∆n−2(x), ∆n−1(x), ∆n+1(x),∆n+2(x)

]

.The posterior is factorised in a Bayesian fashion as follows

P (s|∆n, Ediv) ∝ P (s,∆n, Ediv)

∝ Pt(∆n|s) × Pd(Ediv|s) × Pr(s) (2)

where the indexx has been dropped for clarity. Rather thanconsidering the unknown variablesl(x), v(x) and s(x) asseparate random variables, only one random variables(x)is considered. The values ofl(x) and v(x) can then bedetermined from the estimate ofs(x) according to tables Iand II.

There are two likelihoods associated with the framework,Pt(.) associated with the DFDs of the window andPd(.)associated with the divergence measure. The final term isa prior on the state fields(x). Mathematically, the correctform of this expansion isPt(∆n|Ediv, s) ×Pd(Ediv|s) ×Pr(s).Although it seems likely that some relationship between theDFDs and divergence measure exists, for the purposes of thisframework it assumed that they are statistically independent.

A. Temporal Discontinuity Likelihood

Before the temporal discontinuity likelihood can be intro-duced, the method for calculating the DFDs must be outlined.As has been outlined in Section III-A there are four DFDs tobe estimated described by the four dimensional DFD vector∆n(x). The DFDs are calculated by first compensating themotion of each frame of the window relative to the centralframe. If the image sequence model is the local translationmodel given by

In(x) = In−1(x + dn,n−1(x)) + e(x)

= In+1(x + dn,n+1(x)) + e(x) (3)

wheredh,k is the motion field representing the relative positionof pixels in thekth frame relative to thehth frame, then thefive motion compensated frames (I ′n−2, I ′n−1, I ′n, I ′n+1, I ′n+2)are given by

4

I′n−2(x) = In−2

(

x + dn,n−1 + dn−1,n−2

(

x + dn,n−1

)

)

I′n−1(x) = In−1

(

x + dn,n−1

)

I′n(x) = In(x)

I′n+1(x) = In+1

(

x + dn,n+1

)

I′n+2(x) = In+2

(

x + dn,n+1 + dn+1,n+2

(

x + dn,n+1

)

)

. (4)

Consequently, the four DFDs of the window are given by

∆n−2(x) = I ′n−1(x) − I ′n−2(x)

∆n−1(x) = I ′n(x) − I ′n−1(x)

∆n+1(x) = I ′n(x) − I ′n+1(x)

∆n+2(x) = I ′n+1(x) − I ′n+2(x). (5)

1) Motion Estimation:The motion fields used to performthe motion compensation above are performed as a pre-processto the algorithm. For every frame of the sequence, a backwardand forward motion field must be estimated (i.e. for motion inthe current frame relative to the previous and next frames ofthe sequence respectively). Any motion estimator can be usedto generate the motion fields. The choice of motion estimatoraffects the correct detection and false alarm rate for missingdata and also the PM detection rate. In the tests describedin Section V, the gradient-based algorithm outlined in [13]isused.

2) The Likelihood Expression: The data likelihoodPt(∆n|s) constrains each DFD to be low when a temporaldiscontinuity does not exist. An expression for the likelihoodof a simple temporal discontinuity detector is given by

P (∆(x)|t(x)) ∝ exp−

{

∆(x)2

2σ2e

(

1 − t(x))

+α2

2t(x)

}

(6)

where σ2e is the variance of the model errore(x) and α

acts as a threshold on temporal discontinuities. In the newalgorithm there are four DFDs and the likelihood expressionis formed by multiplying the expressions for four separatetemporal discontinuity detection likelihoods. The likelihoodexpression is given in vector from by

Pt(∆n|s) ∝ exp−3

∑

k=0

{

∆n[k]2

2σ2e

(

1− t[k])

+α2

k

2t[k]

}

(7)

wherek refers to the component index of a vector. For eachcomponent the maximum likelihood condition for a temporaldiscontinuity is

(

∆n[k](x))2

> σ2eα2

k. (8)

The value of the model error variance,σ2e , is determined

by estimating the variance of the DFDs whens(x) = 0. Thethresholdα is allowed to vary for each component of thelikelihood. Theα2 and α3 thresholds for the central DFDs(∆n−1 and∆n+1) are linked to a deterministic threshold onthe DFD,δt. The equivalentα threshold toδt can be derivedfrom equation 8 and is given by

α2 = α3 =

√

δ2t

2σ2e

(9)

(a) Frame with superimposed motion (b) Vector Field Divergence

Fig. 3. The motion of the jacket in the left image is an exampleof PM.The motion field in this region is not smooth and as a result thedivergenceof the motion field in this region has a high absolute value. The dark coloursrepresent high divergence values.

Detection of missing data is made more conservative byusing a lowerα (i.e. for α1 and α4) on the outer DFDs(i.e. ∆n−2 and ∆n+2) than on the central DFDs. The outerthreshold ratio,αr, is defined as the ratio of outer thresholdsα1 andα4 to the inner thresholdsα2 andα3. A typical valueof αr is 0.5.

B. Divergence Likelihood

The divergence of a vector or flow field measures the rate atwhich flow exits a given point of the flow field. The divergenceof a 2D motion field,d(x), is defined as

div(

d(x))

=∂dx(x)

∂x+

∂dy(x)

∂y(10)

wheredx anddy are the horizontal and vertical componentsof the motion field respectively. When PM occurs, the smooth-ness of the motion field is violated and the absolute value ofthe divergence is large (Fig. 3). By finding the divergence ofthe motion fields involved in the motion-compensation of the5-frame window, a divergence measureEdiv(x) can be definedas the sum of the absolute divergences of the four motion fieldsused for motion compensation. That is

Ediv(x) =

n∑

k=n−1

∣

∣div(

dk,k−1(x))∣

∣ +

n+1∑

k=n

∣

∣div(

dk,k+1(x))∣

∣

(11)The divergence likelihood constrainsEdiv to be low for

either missing data or unaffected states. The expression usedfor the likelihood energy in the the framework is

− log(

Pd(Ediv|s))

∝

Λd

(

1 − φ(

Ediv)

)

if s = {0, 1, 2, 4, 6, 8}0 if s = {3, 5, 7, 9, 10,

11, 12, 13, 14, 15}(12)

whereΛd is the weight of the divergence likelihood in theframework and wheres = {3, 5, 7, 9, 10, 11, 12, 13, 14,

15}) is the set of PM states (i.e. l = 2, v = 1) and s = {0,

1, 2, 4, 6, 8}) is the set of unaffected and missing data sites(i.e. l = {0, 1}, v = 0). φ

(

Ediv(x))

is a sigmoidal functiondefined on the divergence measure,Ediv, and is given by

φ(a) =1 + er.et

er.et

×e−r(a−et)

1 + e−r(a−et)(13)

5

0 20 40 60 80 1000

0.2

0.4

0.6

0.8

1r = 0.25r = 0.1r = 0.5

Fig. 4. This figure shows three plots ofφ(a) for a ∈ [0, 100]. In thisexampleet = 50. As r increases the drop in energy aboutet becomes themore abrupt.

where r and et are positive real numbers (See Fig. 4).Effectively, the divergence likelihood acts as a bias towardsthe detection of PM when the divergence measure is large.et

acts as a soft threshold for detecting high divergences.

C. The Smoothness Prior

The prior expression used in the framework is

Pr(s(x)) = Ps(l(x)|L) (14)

which is a spatial smoothness prior onl(x). The smoothnessprior is enforced onl(x) rather thans(x). This reducescomputational complexity since the complexity is proportionalto the number of values a field can hold. Asl(x) is thefield of interest the spatial smoothness of the result is notcompromised.

The prior onl is given by

P (l(x)|L) ∝ exp−{

Λl

∑

y∈Ns(x)

λy

(

1 − u(x,y))

ρ(

l(x), l(y))

}

(15)where Λl is the weight of the prior in the framework (a

value of 1 is commonly used), whereNs(x) is a spatialneighbourhood ofx and whereλy is a weight inversely

proportional to‖x−y‖. ρ(

l(x), l(y))

is the energy penalty for

neighboursx andy having the labelsl(x) andl(y). u(x,y) isa binary edge value between neighbouring pixels which turnsoff smoothness across image edges, allowing sharp transitionsin l(x) across edges [14].u(x,y) is defined on the centralframeIn(x).

1) The PM Bias:Usually the value ofρ(

l(x), l(y))

is 0

for all l(x) = l(y) and 1 for all l(x) 6= l(y). However, inthis framework a bias is introduced intoρ to prevent a partof an “object” being classified as PM and another as missingdata (Fig. 5). This is achieved by increasing the penalty forassigning a site which has a neighbouring PM site as a missingdata site (ρ(1, 2)). The effect is to make blotch detection moreconservative by favouring PM detection at such sites. Theexpression used forρ is

ρ(

l(x), l(y))

=

0 if l(x) = l(y)

K if l(x) = 1 AND l(y) = 2

1 otherwise

(16)

(a) Result without the Missing DataBias

(b) Result with the Missing DataBias

Fig. 5. This figure shows the effect of including the missing data biasin the framework. In the images above a propellor blade undergoing PM ishighlighted by the blue circle. Detected PM is highlighted in red and detectedmissing data is highlighted in green. When the standard smoothness prioris used (i.e. K = 1) , the top part of the blade is incorrectly detected asmissing data. Introducing the PM bias penalises against such a configurationby favouring the detection of PM over missing data. Consequently, the entireblade has been classified as PM when the bias is introduced.

whereK > 1. A value of 5 is used forK in the experimentsperformed in Section V.

D. Solving forl(x)

An estimate forl(x) is found by finding the MAP esti-mate of s(x) (Eq. 2) using the Iterated Conditional Modes(ICM) algorithm [15]. This process is iterated until the resultconverges, with a maximum of 20 iterations. The pixels areupdated using a “checkerboard” scan.

ICM gives a sub-optimal estimate ofs. The convergedestimate represents a local maximum in the posterior PDF.Consequently, a good initialisation ofs is necessary to ensurethat the converged result is close to the global maximum.A deterministic temporal discontinuity detector is used toestimatet(x) and is described by

tk(x) =

{

1 if ∆k(x) > δt

0 otherwise,

k ∈ {n − 2, n − 1, n + 1, n + 2} (17)

where

t(x) = [tn−2(x), tn−1(x), tn+1(x), tn+2(x)].

From the initial estimate oft(x), estimates ofs(x) and l(x)can be found.

1) Multiresolution: A multiresolution scheme [16] is in-corporated into the algorithm. Using mulitresolution results infaster convergence and the state field,s(x), is more likely toconverge to the global maximum of the posterior PDF.

A hierarchical pyramid of state fields of differing resolutionis constructed, with the full resolution field at the bottom levelof the pyramid (level0) and with fields downsampled by afactor of two in each dimension at the next level up in thepyramid. A similar pyramid is constructed for each of theDFDs and for the divergence measure, with all being filteredat each level with a gaussian kernel of variance2.5 beforebeing downsampled.

The algorithm proceeds by initialising the state field,s(x),at the coarsest level of the pyramid using equation 17. A new

6

estimate ofs(x) at the coarsest level (four levels are used)is obtained from the probabilistic framework and the newestimate is then used to initialising the framework at the levelbelow. This process continues untils(x) has been estimatedat full resolution.

An adjustment is made to the prior energy expression whenestimatings(x) at the coarser resolution. The weightλy ismodified to reflect the increased spatial correlation betweena pixel and horizontal and vertical neighbours relative itsdiagonal neighbours. At a leveli of the pyramid, the valueof λy is given by

λy =

{

2i +√

2(2i − 1) for vertical and horizontal neighbours1√2

for diagonal neighbours(18)

V. RESULTS

The main contribution of the proposed algorithm is theintroduction of a joint Markov Random Field (MRF) modelfor blotches and PM (the label fieldl(x)). Pixels can either beclean, a blotch, or PM. Even though blotches can exist at PMsites, such sites cannot be detected as blotches. Effectively, thecorrect detection rate for blotches is compromised in ordertoprevent the false detection of blotches due to PM. Anotherfeature of the algorithm is the application of a smoothnessprior to the label fieldl(x) instead of the full state fields(x). This allows smoothness to be maximised on the fieldof interest (i.e. l(x)) while at the same time minimisingthe computational cost of estimating the spatial smoothnessenergy. This facilitates the introduction of a PM bias, whichpenalises the detection of neighbouring blotch and PM sitesbyadding an energy penaltyK to the assignment of the site as ablotch for every PM site in its neighbourhood (See Fig. 5). Theother significant features of the algorithm are the divergencelikelihood and the outer threshold ratioαr.

This section presents an evaluation of the proposed algo-rithm. It compares the performance of the algorithm to othermissing data detectors, including conventional missing data de-tectors such as SDIp [1] and also the Bornard missing data/PMdetection algorithm. This is achieved by comparing the resultsfor l(x) generated by each algorithm against a ground truthfor blotches. Ground truths for dirt are generated using bothInfra-Red scans and by manual detection of blotches. Thecorrect detection and false alarm rates for missing data inthe sequence are measured and are arranged in the form ofReceiver Operating Characteristics (ROCs), with the controlparameter being the value of the thresholdδt.

However, ROCs cannot give a complete picture of theperformance of the detector. Blotch detectors that modelPathological Motion will perform poorly in terms of correctdetections as they are typically designed to prevent detectionof blotches in regions of motion estimation failure. Ideally,it would be desirable to ignore ground truth blotches in PMregions when estimating the correct detection rates. However,defining an objective and unbiased ground truth for PM isnot a straight forward task. In order to gain a complete under-standing of the performance of the algorithm it is necessaryto

Fig. 6. The “dance” sequence is the first of the ground truth sequences.The movement of the dancer in the foreground is pathological. Its motion isself-occluding which also causes occlusion and uncoveringof the background.

augment the ROC plots with a visual evaluation of the masksgenerated by the algorithm and also examine the detectionrates of PM.

The remainder of this section outlines the experiments usedto evaluate the performance of the algorithm. The algorithmis tested on two sequences with PM for which a ground truthfor blotches was available and also on sequences for whichno ground truth exists. The performance of the proposed al-gorithm is compared to other blotch detectors in Section V-D,including conventional detection algorithms and the Bornardalgorithm. Finally, a discussion of the results is presented.Before the evaluation begins, the two ground truth sequencesare introduced and a description of the experimental procedureis given.

A. Ground Truth Acquisition

The first ground truth sequence is known as the “dance”sequence (See Fig. 6). The motion of the dancer in theforeground is pathological. As the dancer turns, her armsbecome occluded behind the rest of her body. There is alsoocclusion of the background.

The blotch ground truth for this sequence is acquiredusing an Infra-Red (IR) scanner. The Infra-Red scans wereprovided by INA [17] as part of the EU Prestospace project[18]. IR scanners are used to produce a transparency mapof each frame, where the intensity of each pixel in the mapis proportional to the degree of transparency of the pixel. Auseful property of dirt is that it is opaque to IR and as a result isassociated with low intensities in the transparency map. Inthe“dance” sequence a ground truth is estimated for 125 framesof 720×576 pixels. The ground truth is obtained by applyinga threshold of 150 to each 8-bit transparency map.

Using IR scans is a good way to automatically generatea ground truth for dirt. However there are a number ofdrawbacks to using IR scans. IR scans detect dirt even whendirt intensities are similar to the true image intensity. Such dirtcan not be detected by image-based missing data detectors. IRscans will also detect non-impulsive dirt such as line scratches.As a result of these factors, the reported correct detectionratewill be lower than the effective rate.

The second method used to generate a ground truth is tomark blotches in each frame by hand. Acquiring a ground truthin this manner ensures that only perceivable dirt is marked,allowing for a more realistic ground truth. A ground truth for

7

(a) 2 frames from the jacket sequence

(b) Frames with blotches highlighted in white

Fig. 7. The second ground truth sequence is the “jacket” sequence. Asthe person in the sequence removes his jacket, the motion of the jacket ispathological and is an example of non-rigid object motion. This sequence hasa higher blotch frequency than the “dance” sequence.

22 frames of the “jacket” sequence (See Fig. 7) was generatedin this manner. The resolution of this sequence is also720 ×576. The motion of the jacket is an example of the erraticmotion of non-rigid bodies. This sequence also has a higherfrequency of blotches than the “dance” sequence.

B. Experimental Procedure

1) Motion Estimation -:The gradient based motion estima-tor described by Kokaram in [13] is used to generate motionvectors for all the test sequences. The block size of vectorfield is 17 × 17 pixels with a2 pixel horizontal and verticaloverlap.

2) Configuration of the proposed algorithm -:The keyparameter of the algorithm is the deterministic thresholdδt

which is used as the control parameter for the ROC plots. Thechosen values ofδt are{5, 7, 9, 11, 13, 15, 17.5, 20, 25, 30, 35,40}. These values are also used for the equivalent thresholdsin the other missing data detectors used in the comparisonoutlined in Section V-D, ensuring that each detector is detectedover an equivalent range. The model errorσ2

e is estimatedbefore the first stage at each multiresolution level by findingthe variance of the four DFDs whens(x) = 0. The otherrelevant parameters are defined in Table III.

3) ROC plots -:The ROC curves are generated by findingthe average correct detection and false alarm rates over alltheframes of the test sequences for each value ofδt. The missingdata correct detection rate,rc, and false alarm rate,rf , areestimated for each frame (and for eachδt), and are given by

rc =

∑

x bgt(x)bdet(x)∑

x bgt(x)

rf =

∑

x(1 − bgt(x))bdet(x)∑

x(1 − bgt(x))(19)

(a) Validation image

(b) PM/blotch detection

Fig. 8. This figure shows a validation image and a PM/blotch detection forthe “jacket” sequence (correct detections - green, missed detections - blue,false alarms - red). The PM/blotch (red/green) images can beused to indicatewhether or not a missed blotch detection occurs to the pixel being detectedas PM (blue circle).

TABLE III

THE EXPERIMENTAL PARAMETER VALUES FOR THEALGORITHM.

Parameter ValueNumber of Multiresolution Levels 4Number of Iterations at each level 20

α2, α3 See Eq. 9Outer Threshold Ratio,αr 0.5

Divergence Weight,Λd 1Divergence Measure Threshold,et 50

Roll-Off Rate,r 0.5Spatial Smoothness Weight,Λd 1

PM Bias,K 5

wherebgt(x) is the binary ground truth blotch mask for theframe andbdet(x) is the binary output of the blotch detectorunder test. The curve is a parametric plot of corresponding(rc, rf ) pairs.

4) PM rate plots -: Plots of PM rate,rp, against the DFDthreshold δt are evaluated by finding the average relativefrequency of PM in a sequence at each threshold. The PMrate is given by

rp =

∑

x |l(x) == 2|∑

x 1(20)

where PM occurs whenl(x) = 2.

8

(a) A frame from the “dance” sequence (b) Configuration after Deterministic Initialisation(c) Final Field at level 3 (6 iterations in total)

(d) Field after1st iteration at Level 2 (7 iterationsin total)

(e) Final Field at Level 2 (12 iterations) (f) Field after 1st iteration at Level 1 (13 itera-tions)

(g) Final Field at Level 1 (32 iterations) (h) Field after1st iteration at full resolution (33iterations)

(i) Field after final iteration (52 iterations)

Fig. 9. This figure shows intermediate values of the label field (PM - Red, Blotches - Green) at various iterations of the ICMoptimisation process for aframe from the “dance” sequence. The standard parameter setis used (Section V-B) and the value ofδt is 7. Level 0 corresponds to the full resolution; theresolution at level 1 is360 × 288 and so on. At the coarsest resolution (Level 3) the resolution is 90 × 72 pixels.

5) Visual Evaluation -:Two types of image are used for thevisual evaluation. They are validation images and PM/blotchdetections (See Fig. 8). Validation images compare the esti-mated blotch masks to the ground truth and highlight correctdetections (green), missed detections (blue) and false alarms(red). PM/blotch detections display the masks generated bythe proposed algorithm and the Bornard algorithm with PMregions highlighted red and blotches highlighted green.

6) Implementation of the Algorithm:The tests outlined onthe proposed algorithm in this paper utilised a C++ implemen-tation of the algorithm, incorporating the IPP libraries [19]where appropriate. The algorithm was tested on a PC witha 1.4 Giga-Hertz Pentium M processor with 1 Giga-Byte ofRAM.

C. Algorithm Evaluation

Fig. 9 gives an example of the evolution of the label fieldconfiguration for a frame of the “dance” sequence. Detectionof missing data and PM at the coarser resolutions allows largePM and blotch features to be detected. As the resolution atwhich the optimisation occurs increases, the precision of theresult increases, allowing small features to be detected. Ingeneral, convergence at the coarser levels is fast as spatialinformation propagates quickly through the field, but as theresolution increases the speed of convergence decreases. Inthe example shown in Fig. 9, the convergence occurs after sixiterations at both Level3 and Level2. At the finer resolutionsof Level1 and Level0, the maximum iteration limit of twentyiterations is reached. Computation time for this example isapproximately ten seconds.

9

10−6

10−4

10−2

100

0

0.2

0.4

0.6

0.8

1

False Alarm Rate, rf

Cor

rect

Det

ectio

n R

ate,

rc

SDIpsRODMorrisBlotch MRFBornardBlotch/PM MRF

(a) ROC for the “dance” sequence

10−5

10−4

10−3

10−2

10−1

100

0

0.2

0.4

0.6

0.8

1

False Alarm Rate, rf

Cor

rect

Det

ectio

n R

ate,

rc

SDIpsRODMorrisBlotch MRFBornardBlotch/PM MRF

(b) ROC for the “jacket” sequence

Fig. 12. This figure shows the ROC curves for the 6 detectors under test for each test sequence. The standard parameter set is used for the proposedalgorithm. The set up of each detector is described in Section V-B.

10−4

10−3

10−2

0

0.1

0.2

0.3

0.4

0.5

False Alarm Rate, rf

Cor

rect

Det

ectio

n R

ate,

rc

(a) ROC for the “dance” sequence

5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

δt

PM

Det

ectio

n R

ate,

rp

(b) PM rate curve for the “dance”sequence

10−4

10−3

0

0.1

0.2

0.3

0.4

0.5

0.6

False Alarm Rate, rf

Cor

rect

Det

ectio

n R

ate,

rc

(c) ROC for the “jacket” sequence

5 10 15 20 25 30 35 400

0.2

0.4

0.6

0.8

1

δt

PM

Det

ectio

n R

ate,

rp

(d) PM rate curve for the “jacket”sequence

Fig. 10. This figure shows ROC and PM rate plots (rp v δt) for both testsequences obtained from the proposed algorithm using the standard set ofparameters.

Applying the algorithm to all the frames of the test se-quences, gives the ROC and PM rate curves shown in Fig.10. Overall the maximum correct detection rates,rc, for bothsequences are quite low (approximately25% for the “dance”sequence and40% for the “jacket” sequence). However there isalso a low false alarm rate,rf , which does not increase above0.1% (roughly 400 pixels in a720 × 576 image) even at lowthreshold values. The PM rate plots for both test sequences(Fig. 10 (b) and (d)) show that the rate of PM detectionincreases exponentially asδt decreases.

An unusual feature of the ROC plots for the test sequencesis the apparent ceiling on the correct detection rates at lowervalues of the DFD thresholdδt. The expected trend in an ROC

(a) Validation image forδt = 5 (b) Validation image forδt = 7

(c) PM/blotch detection forδt = 5 (d) PM/blotch detection forδt = 7

Fig. 11. This figure illustrates how lowering the DFD threshold δt results ina lower correct detection rate. The blue circle highlights alarge blotch whichis not detected at the lower threshold ((a) and (b) - correct detections in green,missed detections in blue). At the lower threshold this blotch is classified asPM ((c) and (d) - red for PM, green for blotches).

plot of a blotch detector is to have low correct detection andfalse alarm rates at higher threshold values and for the ratesto increase as the value of the threshold drops, tending to anotional correct detection and false alarm rate of100% at azero threshold when every pixel in the image is detected as ablotch. The lowest value of the threshold used therefore givesthe highest false alarm and correct detection rate. Howeverthe ROC curves show that this pattern does not apply to theproposed algorithm. While the expected pattern is followedat high values of the thresholdδt, at lower values of thethreshold a value is reached, below which the correct detectionrate no longer increases and continues to drop off. Althoughthe rate at which temporal discontinuities are detected in

10

(a) Validation image for the Blotch MRF Detector(green - correct detections, red - false alarms, blue- missing data)

(b) Validation image for the proposed algorithm(c) PM/blotch detection for the proposed algo-rithm (red - pm, green - blotches)

Fig. 13. This figure consists of validation images for the Blotch MRF algorithm and the proposed algorithm for a frame fromthe “jacket” sequence. Thevalue of the DFD threshold,δt, is 15. Also shown is the PM/blotch detection for the proposed algorithm. The blue square highlights the region of the imagewhere the proposed algorithm eliminates false alarms caused by Pathological Motion. The yellow circles highlight missed blotch detections in the proposedalgorithm, which have been wrongly detected as PM.

(a) Validation image for the Blotch MRF Detector(green - correct detections, red - false alarms, blue- missing data)

(b) Validation image for the proposed algorithm(c) PM/blotch detection for the proposed algo-rithm (red - pm, green - blotches)

Fig. 14. This figure consists of validation images for the Blotch MRF algorithm and the proposed algorithm for a frame fromthe “dance” sequence, as wellas the corresponding PM/blotch detection for the proposed algorithm. The value of the DFD threshold,δt, is 15. Using the proposed algorithm reduces thenumber of false alarms in the region of the head of the female dancer.

the five frame window continues to increase, an increasingnumber of pixels are classified as PM (See Fig. 11) sincethe likelihood of temporal discontinuities at the same locationbetween multiple frames greatly increases. This is confirmedby the exponentially increasing PM rate at lower thresholdvalues. Consequently, less pixels are detected as missing dataand the correct detection rate decreases. For this detector,the notional false alarm rate and correct detection rate at thezero threshold level is0% as all pixels will be detected asPathological Motion.

D. Comparison with other Missing Data Detectors

The proposed algorithm is compared with five existing miss-ing data detectors. These include four conventional algorithmsand the Bornard Missing Data Detection algorithm [10]. Thefour other algorithms are the SDIp algorithm [2], [13], thesROD detector [4], [12], the Morris algorithm [5], [2] and theBlotch MRF algorithm [6].

1) Detector Configuration:Both the SDIp and sROD algo-rithms have one associated parameterδt and are implementeddirectly from the description of the algorithms. The four prob-

abilistic algorithms are implemented with as similar parametervalues as possible. A multiresolution framework is employedfor each algorithm, with a maximum of twenty iterations perlevel. An eight-pixel first order spatial neighbourhood is usedfor either the blotch field (blotch MRF algorithm), the tem-poral discontinuity field (TDFs) (Morris/Bornard algorithms)or the blotch/PM field (the proposed algorithm). The Bornardalgorithm also employs a two-pixel temporal neighbourhoodlinking the TDFs. Like the proposed algorithm, in the BlotchMRF algorithm a smoothness prior is only enforced on theblotch field and not on the state field.

The value of the spatial smoothness weight in the Morrisand Bornard algorithms is1, and the temporal prior weightof the Bornard algorithm,β3, is −0.1. This reflects the setuprecommended by Bornard in his thesis ([10] Section 5.7.1.2),where the temporal prior weight is roughly one tenth thesize of the spatial weight, with a negative value. The searchradius in the interpretation stage is20 pixels with a temporaldiscontinuity threshold of1. Although less conservative thanthe recommended setup (a radius of38 pixels), PM will beflagged if a temporal discontinuity exists in one of2 × 202π

11

(a) A frame from the “dance” se-quence

(b) A PM/blotch (red/green) detec-tion for the frame (δt = 11)

Fig. 15. This blue circle in the highlights a blotch located near a region ofmotion blur. Due to its location the blotch is detected as PM by the proposedalgorithm.

pixels which is approximately two and a half thousand pixels.For the proposed algorithm, the standard set of parametersdescribed in Section V-B is applied.

2) Comparison with the Conventional Detectors:ROCcurves are plotted for each detector, with the same set of valuesof δt as before (δt = {5, 7, 9, 11, 13, 15, 17.5, 20, 25, 30,35, 40}). Comparison (Fig. 12) of the missing data detectorsshows that both the proposed algorithm and the Bornardalgorithm achieve a significantly reduced false alarm rate whencompared to the four standard missing data detectors, althoughthe correct detection rates are also reduced. At higher valuesof the DFD thresholdδt (i.e. before the maximum correctdetection rate is reached), the false alarm rate of the proposedalgorithm is75% to 90% less for the “dance” sequence and50% to 80% less for the “jacket” sequence. On the other hand,both the Bornard and proposed algorithms have an upper limiton the rate of correct blotch detections. The average maximumcorrect detection rate of the proposed algorithm is two tothree times less than the correct detection rates at the lowestthreshold tested (δt = 5), although the false alarm rate is 30to 50 times lower.

Figures 13 and 14 visually highlight the difference inperformance between the proposed algorithm and the BlotchMRF algorithm, the standard blotch detector most closelyrelated to the proposed algorithm. In the highlighted framesthe proposed algorithm reduces the number of false alarms inthe Pathological Motion regions (i.e. the jacket and the femaledancer), preventing damage to image data. The drop in blotchdetections can mainly be attributed to two factors. Firstlyblotches which are located near objects undergoing PM arelikely to be missed (Fig. 15). This is an intended consequenceof the algorithm design, which is encouraged to maintainimage quality rather than remove the maximum number ofblotches. The second major cause of missed detections occursbecause of a high frequency of blotches over a number offrames. A fundamental assumption made by missing datadetectors is that blotches are rarely located at the same spatiallocation in neighbouring frames. However, if there is a highfrequency of blotches in a sequence, then the probability ofblotches occurring at the same location in neighbouring framesincreases. In such situations, a blotch is likely to be incorrectlyclassified as Pathological Motion.

(a) Validation image for the Bornarddetector (green - correct detections,red - false alarms, blue - missingdata)

(b) PM/blotch detection for theBornard detector (red - pm, green -blotches)

(c) Validation image for the proposedalgorithm

(d) PM/blotch detection for the pro-posed algorithm

Fig. 16. This figures illustrates the superior blotch detection rate of theproposed algorithm over the Bornard algorithm in the “jacket” sequence. Thehighlighted regions of the frame indicate blotches missed by the Bornardalgorithm (image (a)) that are detected by the proposed algorithm (c). Theblotches missed by the Bornard algorithm are detected as PM (b). The valueof δt used to generate the images in this figure is15.

3) Comparison with the Bornard Algorithm:The missingdata ROC curves in Fig. 12 also highlight the variable per-formance of the proposed algorithm when compared with theBornard algorithm. The Bornard algorithm generally performsbetter on the “dance” sequence. At high thresholds the falsealarm rate,rf , of the Bornard algorithm is lower for similarcorrect detection rates,rc. However, at lower threshold valuesthe difference in the false alarm rates decreases and the pro-posed algorithm achieves a higher maximum correct detectionrate. On the other hand, the proposed algorithm is much moreeffective on the “jacket” sequence and achieves a much higherblotch detection rate. The maximum correct detection rateis approximately40%, roughly twice the maximum correctdetection rate of the Bornard algorithm.

The poor correct detection rates of the Bornard algorithmon the “jacket” is clearly visible in the frame from the “jacket”sequence highlighted in Fig. 16. Many of the blotches in theframe are detected as PM by the Bornard algorithm, causedby the high frequency of visible blotches in the sequence. Theproposed algorithm is less prone to missed detections in suchcircumstances.

It is reasonable to conclude that the Bornard detector is moreconservative than the proposed algorithm. The interpretationstage of the Bornard is more conservative than the cumulativeeffect of the divergence likelihood, the PM bias and the outerthreshold ratio. The interpretation stage acts as a “maximumcaution” classifier, rejecting any candidate blotch site ifasingle temporal discontinuity exists within a radius (20 pixels)of the candidate in either of the outer temporal discontinuity

12

(a) From left to right, frames from the “jacket”, “vj”, “birds” and “chopper” sequences.

(b) PM/blotch detections of the four frames for the proposedalgorithm

(c) Frames restored with the Blotch MRF algorithm

(d) Frames restored with the proposed algorithm

Fig. 17. This figure shows frames from four sequences restored using the the Blotch MRF and proposed algorithms as blotch detectors. The highlightedframes show image damage present if the Blotch MRF is used instead of the proposed algorithm.

fields. The ultra cautious nature of the interpretation allows theBornard detector to achieve lower false alarm rates. However,it also results in more missed detections at lower thresholdvalues or when there is a high frequency of blotches in asequence.

On the other hand, the proposed algorithm tries to makean informed decision and as such is less indiscriminate thanthe Bornard algorithm. For a site to be detected as PMrather than a blotch, the proposed algorithm requires eithera temporal discontinuity to exist in at least one outer temporaldiscontinuity field at the same location or spatial propagationfrom neighbouring PM sites by means of the smoothness prior.The divergence likelihood, the PM bias and the outer thresholdare all intended to make the proposed algorithm more likely to

detect PM. The divergence likelihood increases the probabilityof sites with high motion field divergences being detected asPM, the PM bias encourages the spatial propagation of PMstates and the outer threshold ratio increases the likelihood oftemporal discontinuities being detected in the outer fields.

E. Blotch Restoration using the Proposed Algorithm

The critical design criteria of the proposed algorithm is thatit eliminates image damage during missing data treatment. Aprimitive missing data treatment algorithm was implementedthat takes a blotch mask from any detector and interpolatesdetected blotches with the mean motion compensated intensityof the previous and next frames.

13

Fig. 17 compares images restored using both the BlotchMRF algorithm and the proposed algorithm (δt = 15). Framesfrom four sequences are shown. The first is the “jacket”sequence used in the ground truth experiments, in which themotion of the jacket is an example of self-occluding and non-rigid object motion. The second sequence shown is called the“vj” sequence and is an example of repetitive occlusions, astheaircraft propellors appear and disappear in alternative frames.The third sequence is the “birds” sequence. This sequencecontains another example of repetitive occlusion (the bird’swings), however, in this case, the region of PM is movingacross the frame. The final sequence (the “chopper” sequence)contains an example of motion blur. The rotation of the rotorsis also an issue, as the motion estimator employs a purelytranslational motion model.

The four examples shown in Fig. 17 highlight the effective-ness of the proposed algorithm. Using the proposed algorithmprevents significant damage to each of the four images shown.However, the proposed algorithm is not always able to preventall image damage, especially in cases where the region of PMis undergoing a translation (i.e. the motion of an object isthe sum of PM and a non-pathological translation (see Fig.18)). The assumption is made that motion compensation canbe performed accurately, even in regions of PM. Consequently,the algorithm is prone to failure as it involves the motioncompensation of four frames with respect to the central frame.However, it should also be noted that while some imagedamage may occur, any damage will be much less than thedamage caused by performing missing data treatment withoutdetection of PM.

F. Computational Complexity

Estimation of the prior energy and estimation of theMAP state at each pixel is the most computationally in-tensive aspect of the algorithm, amounting to approximately90% of the execution time for a single frame. The up-per bound for the complexity of the ICM optimisationis O(# iterations × # pixels × # states× #neighbours)where the number of states is 16 and the number of pixels in agiven neighbourhood is eight. The actual order of complexityis slightly reduced as the smoothness prior is enforced on thelabel field (three states) rather than the full state field (16states). The number of iterations necessary for convergencealso depends on the number of states and the size of theneighbourhood, with a higher number of states and greaterneighbourhood size requiring more iterations for convergence.From observation of the algorithm, the size of theδt thresholdalso impacts on the number of iterations required. The aver-age number of iterations performs decreases asδt increases.Computation time varies between 5 and 15 seconds per frame,depending on the number iterations performed.

Further reductions in computation would be possible byreducing the complexity of the probabilistic model. For ex-ample, the model could be approximated by reducing thenumber of labels inl(x) to two (i.e. Missing Data or notMissing Data) and by using a mixture model to approximatethe data likelihood distributions. This would also allow aGraph Cuts[20] based segmentation to be performed.

(a) A frame from the “birds” se-quence

(b) Restored frame

Fig. 18. This figure shows an example from the “birds” sequence where theproposed algorithm fails to prevent damage to image data (the highlightedregions). In this example, the object (the bird) undergoingPM is alsoundergoing a translation. This motion causes the algorithmto fail on someframes but not others.

VI. F INAL COMMENTS

This paper has presented a new robust missing data de-tection algorithm that aims to reduce false alarms due toPathological Motion. It uses a probabilistic framework todetect regions of PM, preventing them from being detectedas blotches. A quantitative comparison of the algorithm waspresented, comparing the algorithm against four standardblotch detectors and the Bornard PM detection algorithm.The comparison showed that the proposed algorithm givesa significantly reduced false alarm rate when compared withthe standard detectors. Although a lower false alarm rate canbe obtained with the Bornard algorithm at high thresholds,higher correct detection rates are possible with the proposedalgorithm. A further visual evaluation of the algorithm showshow the algorithm can prevent image damage when used as adetector in a missing data treatment framework.

However the proposed algorithm does not address how theimage is to be restored in PM regions. The missing dataalgorithms outlined in this paper will fail in these regions.Restoration in PM regions has been a topic touched on byRares in [11] in which an alternative metric for detectingblotches in PM was proposed. As temporal reconstruction ofmissing data in PM regions is unreliable, spatial reconstructiontechniques are necessary such as image inpainting [21] ortexture synthesis [22].

The major area of further development in this area wouldbe to integrate the algorithm into a Missing Data Treatmentframework. The algorithm outlined in this paper is presentedas a stand-alone blotch detector which operates independentlyof a missing data interpolation stage. Previous work in MissingData Treatment [23], [8], [24] has shown that integrating thedetection, interpolation and motion estimation stages into asingle Bayesian framework allows for a more visually pleasingrestoration. A new missing data treatment algorithm alongthe lines of the JONDI algorithm [24] using the proposedalgorithm for the detection stage could result in an integratedframework resistant to Pathological Motion.

VII. A CKNOWLEDGEMENTS

This work has been supported by the Irish Research Councilfor Science Engineering and Technology’s Embark initiative,

14

the Adobe corporation, the European Union (EU) Prestospaceproject and the European Marie Curie Transfer of Knowledgeproject Axiom.

REFERENCES

[1] Anil Kokaram. Motion Picture Restoration: Digital Algorithms forArtefact Suppression in Degraded Motion Picture Film and Video,chapter 6. Springer Verlag, 1998.

[2] A. Kokaram, R. Morris, W. J. Fitzgerald, and P. Rayner. Detectionof missing data in image sequences.IEEE Transactions on ImageProcessing, 4:1496–1508, November 1995.

[3] M. J. Nadenau and S. K. Mitra. Blotch and sratch detectionin image se-quences based on rank order differences. In5th International Workshopon Time-Varying Image Processing and Moving Object Recognition,September 1996.

[4] J. Biemond, P. M. B. van Roosmalen, and R. L. Lagendijk. Improvedblotch detection by postprocessing. InIEEE International Conferenceon Acoustics and Signal Processing, pages 3101–3104, March 1999.

[5] R. D. Morris. Image Sequence Distribution using Gibbs Distributions.PhD thesis, Cambridge University, 1995.

[6] B. Kent, A. Kokaram, B. Collis, and S. Robinson. Two layersegmenta-tion for handling pathological motion in degraded post production media.In IEEE Int. Con. on Image Processing (ICIP), Singapore, 2004.

[7] J. Ren and T. Vlachos. Dirt detection for archive film restoration usingan adaptive spatio-temporal approach. InIEE Conference on VisualMedia Production, pages 221–230, November 2005.

[8] Anil Kokaram and Simon Godsill. Mcmc for joint noise reductionand missing data treatment in degraded video.IEEE trans. on SignalProcessing, 50:189–205, February 2002.

[9] A. Rares, M. J. T. Reinders, and J. Biemond. Statistical analysis ofpathological motion areas. InIEE Seminar on Digital Restoration ofFilm and Video Archives, London, UK, 2001.

[10] R. Bornard. Probabilistic Approaches for the Digital Restoration ofTelevision Archives. PhD thesis, Ecole Centrale Paris, 2002.

[11] A. Rares, M. J. T. Reinders, and J. Biemond. Complex event classi-fication in degraded image sequences. InIEEE Int. Conf. on ImageProcessing, Thessaloniki, Greece, 2001.

[12] P. M. B. van Roosmalen.Restoration of Archived Film and Video. PhDthesis, Delft University of Technology, 1999.

[13] Anil Kokaram. Motion Picture Restoration: Digital Algorithms forArtefact Suppression in Degraded Motion Picture Film and Video,chapter 2. Springer Verlag, 1998.

[14] J. Konrad and E. Dubois. Bayesian estimation of motion vector fields.IEEE trans.on Pattern Analysis and Machine Intelligence, 14, September1992.

[15] J. E. Besag. On the statistical analysis of dirty pictures. Journal of theRoyal Statistical Society B, 48:259–302, June 1986.

[16] F. Heitz, P. Perez, and P. Bouthemy. Multiscale minimization of globalenergy functions in some visual recovery problems.CVGIP: ImageUnderstanding, 59:125–134, January 1994.

[17] Institut national de l’audiovisuel. http://www.ina.fr/index.en.html.[18] Prestospace: An integrated solution for audio-visualpreservation and

access. http://www.prestospace.org/.[19] Intel performance primitive (ipp) libraries.

http://www.intel.com/cd/software/products/asmo-na/eng/302910.htm.[20] Y. Boykov and M.-P. Jolly. Interactive graph cuts for optimal boundary

& region segmentation in n-d images. InIEEE Internation Conferenceon Computer Vision, 2001.

[21] A. Rares, M. J. T. Reinders, and J. Biemond. Edge-based imagerestoration. IEEE Transactions on Image Processing, 14:1454–1468,October 2005.

[22] A. A. Efros and T. K. Leung. Texture synthesis by non-paramtericsampling. In IEEE International Conference on Comupter Vission(ICCV), pages 1033–1038, September 1999.

[23] Anil Kokaram. Motion Picture Restoration: Digital Algorithms forArtefact Suppression in Degraded Motion Picture Film and Video,chapter 7. Springer Verlag, 1998.

[24] Anil Kokaram. On missing data treatment for degraded video and framearchives: A survey and a new bayesian approach.IEEE trans. on ImageProcessing, 13:397–414, March 2004.