research article a motion detection algorithm using local...

21
Research Article A Motion Detection Algorithm Using Local Phase Information Aurel A. Lazar, Nikul H. Ukani, and Yiyin Zhou Department of Electrical Engineering, Columbia University, New York, NY 10027, USA Correspondence should be addressed to Aurel A. Lazar; [email protected] Received 15 June 2015; Revised 14 October 2015; Accepted 15 October 2015 Academic Editor: Ezequiel L´ opez-Rubio Copyright © 2016 Aurel A. Lazar et al. is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Previous research demonstrated that global phase alone can be used to faithfully represent visual scenes. Here we provide a reconstruction algorithm by using only local phase information. We also demonstrate that local phase alone can be effectively used to detect local motion. e local phase-based motion detector is akin to models employed to detect motion in biological vision, for example, the Reichardt detector. e local phase-based motion detection algorithm introduced here consists of two building blocks. e first building block measures/evaluates the temporal change of the local phase. e temporal derivative of the local phase is shown to exhibit the structure of a second order Volterra kernel with two normalized inputs. We provide an efficient, FFT- based algorithm for implementing the change of the local phase. e second processing building block implements the detector; it compares the maximum of the Radon transform of the local phase derivative with a chosen threshold. We demonstrate examples of applying the local phase-based motion detection algorithm on several video sequences. We also show how the locally detected motion can be used for segmenting moving objects in video scenes and compare our local phase-based algorithm to segmentation achieved with a widely used optic flow algorithm. 1. Introduction Following Marr, the design of an information processing system can be approached on multiple levels [1]. Figure 1 illustrates two levels of abstraction, namely, the algorithmic level and the physical circuit level. On the algorithmic level, one studies procedurally how the information is processed independently of the physical realization. e circuit level concerns the actual realization of the algorithm in physical hardware, for example, a biological neural circuit or silicon circuits in a digital signal processor. In this paper, we put forth a simple motion detection algorithm that is inspired by motion detection models of biological visual systems (in vivo neural circuit) and provide an efficient realization that can easily be implemented on commodity (in silico) DSP chips. Visual motion detection is critical to the survival of animals. Many biological visual systems have evolved highly efficient/effective neural circuits to detect visual motion. Motion detection is performed in parallel with other visual coding circuits and starts already in the early stages of visual processing. In the retina of vertebrates, it is known that at least three types of Direction-Selective Ganglion Cells (DSGC) are responsible for signaling visual motion at this early stage [2]. In flies, direction-selective neurons are found in the optic lobe, 3 synapses away from the photoreceptors [3]. e small number of synapses between photoreceptors and direction-selective neurons suggests that the processing involved in motion detection is not highly complex but still very effective. In addition, the biological motion detection circuits are organized in a highly parallel way to enable fast, concurrent computation of motion. It is also interesting to note that the early stages of motion detection are carried out largely in the absence of spiking neurons, indicating that initial stages of motion detection are preferably performed in the “analog” domain. Taking advantage of continuous time processing may be critical for quickly processing motion since motion intrinsically elicits fast and large changes in the intensity levels, that is, large amounts of data under stringent time constraints. Modern, computer-based motion detection algorithms oſten employ optic flow techniques to estimate spatial changes in consecutive image frames [4, 5]. Although, oſten Hindawi Publishing Corporation Computational Intelligence and Neuroscience Volume 2016, Article ID 7915245, 20 pages http://dx.doi.org/10.1155/2016/7915245

Upload: others

Post on 22-Aug-2020

11 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Research ArticleA Motion Detection Algorithm Using Local Phase Information

Aurel A Lazar Nikul H Ukani and Yiyin Zhou

Department of Electrical Engineering Columbia University New York NY 10027 USA

Correspondence should be addressed to Aurel A Lazar aureleecolumbiaedu

Received 15 June 2015 Revised 14 October 2015 Accepted 15 October 2015

Academic Editor Ezequiel Lopez-Rubio

Copyright copy 2016 Aurel A Lazar et al This is an open access article distributed under the Creative Commons Attribution Licensewhich permits unrestricted use distribution and reproduction in any medium provided the original work is properly cited

Previous research demonstrated that global phase alone can be used to faithfully represent visual scenes Here we provide areconstruction algorithm by using only local phase information We also demonstrate that local phase alone can be effectively usedto detect local motion The local phase-based motion detector is akin to models employed to detect motion in biological visionfor example the Reichardt detector The local phase-based motion detection algorithm introduced here consists of two buildingblocks The first building block measuresevaluates the temporal change of the local phase The temporal derivative of the localphase is shown to exhibit the structure of a second order Volterra kernel with two normalized inputs We provide an efficient FFT-based algorithm for implementing the change of the local phase The second processing building block implements the detector itcompares the maximum of the Radon transform of the local phase derivative with a chosen threshold We demonstrate examplesof applying the local phase-based motion detection algorithm on several video sequences We also show how the locally detectedmotion can be used for segmenting moving objects in video scenes and compare our local phase-based algorithm to segmentationachieved with a widely used optic flow algorithm

1 Introduction

Following Marr the design of an information processingsystem can be approached on multiple levels [1] Figure 1illustrates two levels of abstraction namely the algorithmiclevel and the physical circuit level On the algorithmic levelone studies procedurally how the information is processedindependently of the physical realization The circuit levelconcerns the actual realization of the algorithm in physicalhardware for example a biological neural circuit or siliconcircuits in a digital signal processor In this paper we putforth a simple motion detection algorithm that is inspired bymotion detection models of biological visual systems (in vivoneural circuit) and provide an efficient realization that caneasily be implemented on commodity (in silico) DSP chips

Visual motion detection is critical to the survival ofanimals Many biological visual systems have evolved highlyefficienteffective neural circuits to detect visual motionMotion detection is performed in parallel with other visualcoding circuits and starts already in the early stages of visualprocessing In the retina of vertebrates it is known that at least

three types of Direction-Selective Ganglion Cells (DSGC) areresponsible for signaling visual motion at this early stage [2]In flies direction-selective neurons are found in the opticlobe 3 synapses away from the photoreceptors [3]

The small number of synapses between photoreceptorsand direction-selective neurons suggests that the processinginvolved in motion detection is not highly complex but stillvery effective In addition the biological motion detectioncircuits are organized in a highly parallel way to enable fastconcurrent computation of motion It is also interesting tonote that the early stages of motion detection are carriedout largely in the absence of spiking neurons indicating thatinitial stages of motion detection are preferably performed inthe ldquoanalogrdquo domain Taking advantage of continuous timeprocessing may be critical for quickly processing motionsince motion intrinsically elicits fast and large changes in theintensity levels that is large amounts of data under stringenttime constraints

Modern computer-based motion detection algorithmsoften employ optic flow techniques to estimate spatialchanges in consecutive image frames [4 5] Although often

Hindawi Publishing CorporationComputational Intelligence and NeuroscienceVolume 2016 Article ID 7915245 20 pageshttpdxdoiorg10115520167915245

2 Computational Intelligence and Neuroscience

Algorithm

In vivo neural circuit

In silicoDSP chip

Figure 1 Algorithms can have different physical realizations Forexample an algorithm can be implemented by neural circuits in abiological system Alternately it can be implemented on a digitalsignal processor Algorithms direct the implementation on thephysical layer Conversely biological neural circuits inspire newalgorithmic designs which can in turn be expressed and improvedupon by a realization in silico

time optic flow estimation algorithms produce accurateresults the computational demand to perform many of thesealgorithms is too high for real-time implementation

Several models for biological motion detection are avail-able and their architecture is quite simple [6] The Reichardtmotion detector [7] was thought to be the underlyingmodel for motion detection in insects [8] The model isbased on a correlation method to extract motion inducedby spatiotemporal information patterns of light intensityTherefore it relies on a correlationmultiplication operationA second model is the motion energy detector [9] It usesspatiotemporal separable filters and a squaring nonlinearityto compute motion energy and it was shown to be equivalentto the Reichardt motion detector Earlier work in the rabbitretina was the foundation to the Barlow-Levick model [10]of motion detection The model relies on inhibition tocompensate motion in the null direction

In this paper we provide an alternative motion detectionalgorithm based on local phase information of the visualscene Similar to mechanisms in other biological models itoperates in continuous time and in parallel Moreover themotion detection algorithm we propose can be efficientlyimplemented on parallel hardware This is again similar tothe properties of biological motion detection systems Ratherthan focusing on velocity of motion we focus on localizationthat is where the motion occurs in the visual field as well asthe direction of motion

It has been shown that images can be represented bytheir global phase alone [11] Here we provide a reconstruc-tion algorithm of visual scenes by only using local phaseinformation thereby demonstrating the spectrum of therepresentational capability of phase information

The Fourier shift property clearly suggests the relation-ship between the global shift of an image and the global phaseshift in the frequency domain We elevate this relationshipby computing the change of local phase to indicate motionthat appears locally in the visual scene The local phases arecomputed using window functions that tile the visual fieldwith overlapping segments making it amenable for a highlyparallel implementation In addition we propose a Radon-transform-based motion detection index on the change of

local phases for the readout of the relation between localphases and motion

Interestingly phase information has been largely ignoredin the field of linear signal processing and for good reasonPhase-based processing is intrinsically non-linear Recentresearches however showed that phase information canbe smartly employed in speech processing [12] and visualprocessing [13] For example spatial phase in an image isindicative of local features such as edges when consideringphase congruency [14] The role of spatial phase in compu-tational and biological vision emergence of visual illusionsandpattern recognition is discussed in [15] Togetherwith ourresult for motion detection these studies suggest that phaseinformation has a great potential for achieving efficient visualsignal processing

This paper is organized as follows In Section 2 weshow that local phase information can be used to faithfullyrepresent visual information in the absence of amplitudeinformation In Section 3 we develop a simple local phase-based algorithm to detectmotion in visual scenes and providean efficient way for its implementation We then provideexamples and applications of the proposed motion detectionalgorithm to motion segmentation We also compare ourresults to those obtained using motion detection algorithmsbased on optic flow techniques in Section 4 Finally wesummarize our results in Section 5

2 Representation of Visual Scenes UsingPhase Information

Theuse of complex valued transforms is widespread for bothrepresenting and processing images When represented inpolar coordinates the output of a complex valued transformof a signal can be split into amplitude and phase In thissection we define two types of phases of an image globalphase and local phase We then argue that both types ofphases can faithfully represent an image and we provide areconstruction algorithm that recovers the image from localphase information This indicates that phase informationalone can largely represent an image or video signal

It will become clear in the sections that follow that theuse of phase for representing of image and video infor-mation leads to efficient ways to implement certain typesof imagevideo processing algorithms for example motiondetection algorithms

21 The Global Phase of Images Recall that the Fouriertransform of a real valued image 119906 = 119906(119909 119910) (119909 119910) isin R2is given by

(120596119909 120596119910) = int

R2119906 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910 (1)

with (120596119909 120596119910) isin C and (120596

119909 120596119910) isin R2

In polar coordinates the Fourier transform of 119906 can beexpressed as

(120596119909 120596119910) = (120596

119909 120596119910) 119890119895(120596119909120596119910) (2)

Computational Intelligence and Neuroscience 3

where (120596119909 120596119910) isin R is the amplitude and

120601(120596119909 120596119910) is the

phase of the Fourier transform of 119906

Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906

It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase

22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)

119880(120596119909 120596119910 1199090 1199100) = int

R2119906 (119909 119910)119908 (119909 minus 119909

0 119910 minus 119910

0)

sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910

(3)

where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909

0 1199100) isin R2 Typical choices of window

functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis

Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as

119880(120596119909 120596119910 1199090 1199100) = 119860 (120596

119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)

where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596

119909 120596119910

1199090 1199100) is the phase of the STFT

Definition 2 The amplitude of the STFT of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906

Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596

119909 120596119910 1199090 1199100) can be equivalently viewed as the

response of a complex-valued Gabor receptive field

ℎ (119909 119910) = 119890minus((119909minus119909

0)2+(119910minus119910

0)2)21205902

119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)

to 119906In this case the window of the STFT clearly is given by

119908 (119909 119910) = 119890minus(1199092+1199102)21205902

(6)

Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]

23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]

Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]

While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve

Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =

119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs

Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (7)

where

119890119897119909119897119910

= exp(119895119897119909

Ω119909

119871119909

119909) exp(119895119897119910

Ω119910

119871119910

119910)

119897119909= minus119871119909 119871

119909 119897119910= minus119871119910 119871

119910

(8)

are the set of basis functions ofHΩ119909 119871119909are the bandwidth

and the order respectively of the space in the 119909 dimensionand Ω

119910 119871119910are the bandwidth and the order respectively of

the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner

product [20 21]

⟨1199061 1199062⟩ = int

[0119879119909]times[0119879

119910]

1199061(119909 119910) 119906

2(119909 119910)119889119909 119889119910 (9)

4 Computational Intelligence and Neuroscience

where 119879119909= 2120587119871

119909Ω119909 119879119910= 2120587119871

119910Ω119910are the period of the

space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields

ℎ119896119897119898119899

(119909 119910) = T119896119897(119908 (119909 119910) 119890

minus119895(120596119909119898119909+120596119910119899119910)) (10)

where T119896119897is the translation operator with T

119896119897119906 = 119906(119909 minus

1198961198870 119910 minus 119897119887

0) 119896 119897 isin Z 119887

0gt 0 0 le 119896119887

0le 119879119909 and 0 le

1198971198870le 119879119910 and 120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960 119898 119899 isin Z 120596

0gt 0

minusΩ119909le 119898120596

0le Ω119909 and minusΩ

119910le 1198991205960le Ω119910 The responses of

the Gabor receptive fields to the input 119906(119909 119910) are given by

int

R2119906 (119909 119910) ℎ

119896119897119898119899(119909 119910) 119889119909 119889119910 = int

R2119906 (119909 119910)

sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887

0) 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870))119889119909 119889119910

= 119860119896119897119898119899

119890119895120601119896119897119898119899

(11)

where 119860119896119897119898119899

ge 0 119860119896119897119898119899

isin R is the local amplitude and120601119896119897119898119899

isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860

119896119897119898119899

(12)

Since 119860119896119897119898119899

isin R we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot cos (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 119860119896119897119898119899

(13)

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 0

(14)

Note that119908(119909 minus 1198961198870 119910 minus 119897119887

0) sin(120596

119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) +

120601119896119897119898119899

) is a real-valued Gabor receptive field with a preferredphase at 120601

119896119897119898119899+ 1205872 minus 120596

119909119898

1198961198870minus 120596119910119899

1198971198870

Remark 3 Assuming that the local phase information 120601119896119897119898119899

is obtained via measurements that is filtering the image 119906

with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions

119908 (119909 minus 1198961198870 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

(15)

where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le

119879119909 0 le 119897119887

0le 119879119910 minusΩ119909le 119898120596

0le Ω119909 minusΩ119910le 1198991205960le Ω119910

We are now in the position to provide a reconstructionalgorithm of the image from phase 120601

119896119897119898119899 (119896 119897 119898 119899) isin I

Lemma 4 119906 can be reconstructed from 120601119896119897119898119899

(119896 119897 119898 119899) isin Ias

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (16)

with

Φ119888 = 0 (17)

whereΦ is a matrix whose 119901th row and 119902th column entry are

[Φ]119901119902

= int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910

(18)

Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+

119871119910+ 1) 119888 is a vector of the form

119888 = [119888minus119871119909minus119871119910

119888minus119871119909minus119871119910+1 119888

minus119871119909119871119910

119888minus119871119909+1minus119871

119910

119888minus119871119909+1minus119871

119910+1 119888

minus119871119909+1119871119910

119888119871119909minus119871119910

119888119871119909minus119871119910+1

119888119871119909119871119910

]

119879

(19)

that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871

119909+ 1)(2119871

119910+ 1) minus 1 where 119873 is the number of phase

measurements

Proof Substituting (7) into (14) we obtain119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910 = 0

(20)

for all 119896 119897 119898 119899 isin Z Therefore we have

Φ119888 = 0 (21)

inferring that 119888 is in the null space ofΦIf 119873 lt (2119871

119909+ 1)(2119871

119910+ 1) minus 1 it follows from the rank-

nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0

Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int

R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)

Computational Intelligence and Neuroscience 5

(a) (b) (c)

Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error

3 Visual Motion Detection fromPhase Information

In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field

31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0) (22)

where

119904119909(119905) = int

119905

0

V119909(119904) 119889119904

119904119910(119905) = int

119905

0

V119910(119904) 119889119904

(23)

are the total length of translation at time 119905 in each dimen-sion and V

119909(119905) and V

119910(119905) are the corresponding instanta-

neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following

Lemma 6 The change (derivative) of the global phase is givenby

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(24)

where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global

phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition

Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier

transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have

(F119906 (sdot sdot 119905)) (120596119909 120596119910)

= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596

119910119904119910(119905))

(25)

For a certain frequency component (120596119909 120596119910) the change of its

global phase over time amounts to

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909

119889119904119909(119905)

119889119905

minus 120596119910

119889119904119910(119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(26)

Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector

32 The Change of Local Phase

321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field

Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 2: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

2 Computational Intelligence and Neuroscience

Algorithm

In vivo neural circuit

In silicoDSP chip

Figure 1 Algorithms can have different physical realizations Forexample an algorithm can be implemented by neural circuits in abiological system Alternately it can be implemented on a digitalsignal processor Algorithms direct the implementation on thephysical layer Conversely biological neural circuits inspire newalgorithmic designs which can in turn be expressed and improvedupon by a realization in silico

time optic flow estimation algorithms produce accurateresults the computational demand to perform many of thesealgorithms is too high for real-time implementation

Several models for biological motion detection are avail-able and their architecture is quite simple [6] The Reichardtmotion detector [7] was thought to be the underlyingmodel for motion detection in insects [8] The model isbased on a correlation method to extract motion inducedby spatiotemporal information patterns of light intensityTherefore it relies on a correlationmultiplication operationA second model is the motion energy detector [9] It usesspatiotemporal separable filters and a squaring nonlinearityto compute motion energy and it was shown to be equivalentto the Reichardt motion detector Earlier work in the rabbitretina was the foundation to the Barlow-Levick model [10]of motion detection The model relies on inhibition tocompensate motion in the null direction

In this paper we provide an alternative motion detectionalgorithm based on local phase information of the visualscene Similar to mechanisms in other biological models itoperates in continuous time and in parallel Moreover themotion detection algorithm we propose can be efficientlyimplemented on parallel hardware This is again similar tothe properties of biological motion detection systems Ratherthan focusing on velocity of motion we focus on localizationthat is where the motion occurs in the visual field as well asthe direction of motion

It has been shown that images can be represented bytheir global phase alone [11] Here we provide a reconstruc-tion algorithm of visual scenes by only using local phaseinformation thereby demonstrating the spectrum of therepresentational capability of phase information

The Fourier shift property clearly suggests the relation-ship between the global shift of an image and the global phaseshift in the frequency domain We elevate this relationshipby computing the change of local phase to indicate motionthat appears locally in the visual scene The local phases arecomputed using window functions that tile the visual fieldwith overlapping segments making it amenable for a highlyparallel implementation In addition we propose a Radon-transform-based motion detection index on the change of

local phases for the readout of the relation between localphases and motion

Interestingly phase information has been largely ignoredin the field of linear signal processing and for good reasonPhase-based processing is intrinsically non-linear Recentresearches however showed that phase information canbe smartly employed in speech processing [12] and visualprocessing [13] For example spatial phase in an image isindicative of local features such as edges when consideringphase congruency [14] The role of spatial phase in compu-tational and biological vision emergence of visual illusionsandpattern recognition is discussed in [15] Togetherwith ourresult for motion detection these studies suggest that phaseinformation has a great potential for achieving efficient visualsignal processing

This paper is organized as follows In Section 2 weshow that local phase information can be used to faithfullyrepresent visual information in the absence of amplitudeinformation In Section 3 we develop a simple local phase-based algorithm to detectmotion in visual scenes and providean efficient way for its implementation We then provideexamples and applications of the proposed motion detectionalgorithm to motion segmentation We also compare ourresults to those obtained using motion detection algorithmsbased on optic flow techniques in Section 4 Finally wesummarize our results in Section 5

2 Representation of Visual Scenes UsingPhase Information

Theuse of complex valued transforms is widespread for bothrepresenting and processing images When represented inpolar coordinates the output of a complex valued transformof a signal can be split into amplitude and phase In thissection we define two types of phases of an image globalphase and local phase We then argue that both types ofphases can faithfully represent an image and we provide areconstruction algorithm that recovers the image from localphase information This indicates that phase informationalone can largely represent an image or video signal

It will become clear in the sections that follow that theuse of phase for representing of image and video infor-mation leads to efficient ways to implement certain typesof imagevideo processing algorithms for example motiondetection algorithms

21 The Global Phase of Images Recall that the Fouriertransform of a real valued image 119906 = 119906(119909 119910) (119909 119910) isin R2is given by

(120596119909 120596119910) = int

R2119906 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910 (1)

with (120596119909 120596119910) isin C and (120596

119909 120596119910) isin R2

In polar coordinates the Fourier transform of 119906 can beexpressed as

(120596119909 120596119910) = (120596

119909 120596119910) 119890119895(120596119909120596119910) (2)

Computational Intelligence and Neuroscience 3

where (120596119909 120596119910) isin R is the amplitude and

120601(120596119909 120596119910) is the

phase of the Fourier transform of 119906

Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906

It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase

22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)

119880(120596119909 120596119910 1199090 1199100) = int

R2119906 (119909 119910)119908 (119909 minus 119909

0 119910 minus 119910

0)

sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910

(3)

where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909

0 1199100) isin R2 Typical choices of window

functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis

Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as

119880(120596119909 120596119910 1199090 1199100) = 119860 (120596

119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)

where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596

119909 120596119910

1199090 1199100) is the phase of the STFT

Definition 2 The amplitude of the STFT of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906

Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596

119909 120596119910 1199090 1199100) can be equivalently viewed as the

response of a complex-valued Gabor receptive field

ℎ (119909 119910) = 119890minus((119909minus119909

0)2+(119910minus119910

0)2)21205902

119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)

to 119906In this case the window of the STFT clearly is given by

119908 (119909 119910) = 119890minus(1199092+1199102)21205902

(6)

Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]

23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]

Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]

While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve

Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =

119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs

Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (7)

where

119890119897119909119897119910

= exp(119895119897119909

Ω119909

119871119909

119909) exp(119895119897119910

Ω119910

119871119910

119910)

119897119909= minus119871119909 119871

119909 119897119910= minus119871119910 119871

119910

(8)

are the set of basis functions ofHΩ119909 119871119909are the bandwidth

and the order respectively of the space in the 119909 dimensionand Ω

119910 119871119910are the bandwidth and the order respectively of

the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner

product [20 21]

⟨1199061 1199062⟩ = int

[0119879119909]times[0119879

119910]

1199061(119909 119910) 119906

2(119909 119910)119889119909 119889119910 (9)

4 Computational Intelligence and Neuroscience

where 119879119909= 2120587119871

119909Ω119909 119879119910= 2120587119871

119910Ω119910are the period of the

space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields

ℎ119896119897119898119899

(119909 119910) = T119896119897(119908 (119909 119910) 119890

minus119895(120596119909119898119909+120596119910119899119910)) (10)

where T119896119897is the translation operator with T

119896119897119906 = 119906(119909 minus

1198961198870 119910 minus 119897119887

0) 119896 119897 isin Z 119887

0gt 0 0 le 119896119887

0le 119879119909 and 0 le

1198971198870le 119879119910 and 120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960 119898 119899 isin Z 120596

0gt 0

minusΩ119909le 119898120596

0le Ω119909 and minusΩ

119910le 1198991205960le Ω119910 The responses of

the Gabor receptive fields to the input 119906(119909 119910) are given by

int

R2119906 (119909 119910) ℎ

119896119897119898119899(119909 119910) 119889119909 119889119910 = int

R2119906 (119909 119910)

sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887

0) 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870))119889119909 119889119910

= 119860119896119897119898119899

119890119895120601119896119897119898119899

(11)

where 119860119896119897119898119899

ge 0 119860119896119897119898119899

isin R is the local amplitude and120601119896119897119898119899

isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860

119896119897119898119899

(12)

Since 119860119896119897119898119899

isin R we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot cos (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 119860119896119897119898119899

(13)

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 0

(14)

Note that119908(119909 minus 1198961198870 119910 minus 119897119887

0) sin(120596

119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) +

120601119896119897119898119899

) is a real-valued Gabor receptive field with a preferredphase at 120601

119896119897119898119899+ 1205872 minus 120596

119909119898

1198961198870minus 120596119910119899

1198971198870

Remark 3 Assuming that the local phase information 120601119896119897119898119899

is obtained via measurements that is filtering the image 119906

with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions

119908 (119909 minus 1198961198870 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

(15)

where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le

119879119909 0 le 119897119887

0le 119879119910 minusΩ119909le 119898120596

0le Ω119909 minusΩ119910le 1198991205960le Ω119910

We are now in the position to provide a reconstructionalgorithm of the image from phase 120601

119896119897119898119899 (119896 119897 119898 119899) isin I

Lemma 4 119906 can be reconstructed from 120601119896119897119898119899

(119896 119897 119898 119899) isin Ias

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (16)

with

Φ119888 = 0 (17)

whereΦ is a matrix whose 119901th row and 119902th column entry are

[Φ]119901119902

= int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910

(18)

Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+

119871119910+ 1) 119888 is a vector of the form

119888 = [119888minus119871119909minus119871119910

119888minus119871119909minus119871119910+1 119888

minus119871119909119871119910

119888minus119871119909+1minus119871

119910

119888minus119871119909+1minus119871

119910+1 119888

minus119871119909+1119871119910

119888119871119909minus119871119910

119888119871119909minus119871119910+1

119888119871119909119871119910

]

119879

(19)

that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871

119909+ 1)(2119871

119910+ 1) minus 1 where 119873 is the number of phase

measurements

Proof Substituting (7) into (14) we obtain119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910 = 0

(20)

for all 119896 119897 119898 119899 isin Z Therefore we have

Φ119888 = 0 (21)

inferring that 119888 is in the null space ofΦIf 119873 lt (2119871

119909+ 1)(2119871

119910+ 1) minus 1 it follows from the rank-

nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0

Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int

R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)

Computational Intelligence and Neuroscience 5

(a) (b) (c)

Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error

3 Visual Motion Detection fromPhase Information

In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field

31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0) (22)

where

119904119909(119905) = int

119905

0

V119909(119904) 119889119904

119904119910(119905) = int

119905

0

V119910(119904) 119889119904

(23)

are the total length of translation at time 119905 in each dimen-sion and V

119909(119905) and V

119910(119905) are the corresponding instanta-

neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following

Lemma 6 The change (derivative) of the global phase is givenby

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(24)

where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global

phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition

Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier

transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have

(F119906 (sdot sdot 119905)) (120596119909 120596119910)

= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596

119910119904119910(119905))

(25)

For a certain frequency component (120596119909 120596119910) the change of its

global phase over time amounts to

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909

119889119904119909(119905)

119889119905

minus 120596119910

119889119904119910(119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(26)

Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector

32 The Change of Local Phase

321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field

Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 3: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 3

where (120596119909 120596119910) isin R is the amplitude and

120601(120596119909 120596119910) is the

phase of the Fourier transform of 119906

Definition 1 The amplitude of the Fourier transform of animage 119906 = 119906(119909 119910) (119909 119910) isin R2 is called the global amplitudeof 119906 The phase of the Fourier transform of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the global phase of 119906

It is known that the global phase of an image plays animportant role in the representation of natural images [11] Aclassic example is to take two images and to exchange theirglobal phases before their reconstruction using the inverseFourier transform The resulting images are slightly smearedbut largely reflect the information contained in the globalphase

22The Local Phase of Images The global phase indicates theoffset of sinusoids of different frequencies contained in theentire image However it is not intuitive to relate the globalphase to local image features such as edges and their positionin an image To study these local features it is necessary tomodify the Fourier transform such that it reflects propertiesof a restricted region of an image It is natural to consider theShort-Time (-Space) Fourier Transform (STFT)

119880(120596119909 120596119910 1199090 1199100) = int

R2119906 (119909 119910)119908 (119909 minus 119909

0 119910 minus 119910

0)

sdot 119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100))119889119909 119889119910

(3)

where 119908 = 119908(119909 119910) (119909 119910) isin R2 is a real valued windowfunction centered at (119909

0 1199100) isin R2 Typical choices of window

functions include the Hann window and the Gaussian win-dow The (effectively) finite support of the window restrictsthe Fourier transform to local image analysis

Similarly to the Fourier transform the STFT can beexpressed in polar coordinates as

119880(120596119909 120596119910 1199090 1199100) = 119860 (120596

119909 120596119910 1199090 1199100) 119890119895120601(12059611990912059611991011990901199100) (4)

where 119860(120596119909 120596119910 1199090 1199100) isin R is the amplitude and 120601(120596

119909 120596119910

1199090 1199100) is the phase of the STFT

Definition 2 The amplitude of the STFT of an image 119906 =

119906(119909 119910) (119909 119910) isin R2 is called the local amplitude of 119906 Thephase of the STFT of an image 119906 = 119906(119909 119910) (119909 119910) isin R2 iscalled the local phase of 119906

Note that when 119908 is a Gaussian window the STFT of 119906evaluated at (120596

119909 120596119910 1199090 1199100) can be equivalently viewed as the

response of a complex-valued Gabor receptive field

ℎ (119909 119910) = 119890minus((119909minus119909

0)2+(119910minus119910

0)2)21205902

119890minus119895(120596119909(119909minus1199090)+120596119910(119910minus1199100)) (5)

to 119906In this case the window of the STFT clearly is given by

119908 (119909 119910) = 119890minus(1199092+1199102)21205902

(6)

Therefore the STFT can be realized by an ensemble of Gaborreceptive fields that are common in modeling simple andcomplex cells (neurons) in the primary visual cortex [16]

23 Reconstruction of Images from Local Phase Amplitudeand phase can be interpreted as measurementsprojectionsof images that are indicative of their information contentClassically when both the global amplitude and phase areknown it is straightforward to reconstruct the image Thereconstruction calls for computing the inverse Fourier trans-form given the global amplitude and phase Similarly whenusing local amplitude and phase if the sampling functionsform a basis or frame in a space of images the reconstructionis provided by the formalism of wavelet theory for exampleusing Gabor wavelets [17]

Amplitude or phase represents partial informationextracted from visual scenesThey are obtained via nonlinearsampling that is a nonlinear operation for extracting theamplitude andphase information from imagesThenonlinearoperation makes reconstruction from either amplitude orphase alone difficult Earlier studies and recent developmentin solving quadratic constraints however suggest that it ispossible to reconstruct images from global or local amplitudeinformation [18 19]

While computing the amplitude requires a second order(quadratic) nonlinearity computing the phase calls for higherorder nonlinear operators (Volterra kernels) It is possiblehowever to reconstruct up to a constant scale an image fromits global phase information alone without explicitly usingthe amplitude information An algorithm was provided forsolving this problem in the discrete signal processing domainin [11]The algorithm smartly avoids using the inverse tangentfunction by reformulating the phase measurements as a set oflinear equations that are easy to solve

Using a similar argument we demonstrate in the follow-ing that up to a constant scale a bandlimited signal 119906 =

119906(119909 119910) (119909 119910) isin R2 can be reconstructed from its local phasealone We first formulate the encoding of an image 119906 by localphase using the Gabor receptive fields as a special case Itis straightforward then to formulate the problem with localphase computed from other types of STFTs

Formally we consider an image 119906 = 119906(119909 119910) on thedomain R2 to be an element of a space of trigonometricpolynomialsH of the form

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (7)

where

119890119897119909119897119910

= exp(119895119897119909

Ω119909

119871119909

119909) exp(119895119897119910

Ω119910

119871119910

119910)

119897119909= minus119871119909 119871

119909 119897119910= minus119871119910 119871

119910

(8)

are the set of basis functions ofHΩ119909 119871119909are the bandwidth

and the order respectively of the space in the 119909 dimensionand Ω

119910 119871119910are the bandwidth and the order respectively of

the space in the 119910 dimensionH is a Reproducing Kernel Hilbert Space with inner

product [20 21]

⟨1199061 1199062⟩ = int

[0119879119909]times[0119879

119910]

1199061(119909 119910) 119906

2(119909 119910)119889119909 119889119910 (9)

4 Computational Intelligence and Neuroscience

where 119879119909= 2120587119871

119909Ω119909 119879119910= 2120587119871

119910Ω119910are the period of the

space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields

ℎ119896119897119898119899

(119909 119910) = T119896119897(119908 (119909 119910) 119890

minus119895(120596119909119898119909+120596119910119899119910)) (10)

where T119896119897is the translation operator with T

119896119897119906 = 119906(119909 minus

1198961198870 119910 minus 119897119887

0) 119896 119897 isin Z 119887

0gt 0 0 le 119896119887

0le 119879119909 and 0 le

1198971198870le 119879119910 and 120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960 119898 119899 isin Z 120596

0gt 0

minusΩ119909le 119898120596

0le Ω119909 and minusΩ

119910le 1198991205960le Ω119910 The responses of

the Gabor receptive fields to the input 119906(119909 119910) are given by

int

R2119906 (119909 119910) ℎ

119896119897119898119899(119909 119910) 119889119909 119889119910 = int

R2119906 (119909 119910)

sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887

0) 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870))119889119909 119889119910

= 119860119896119897119898119899

119890119895120601119896119897119898119899

(11)

where 119860119896119897119898119899

ge 0 119860119896119897119898119899

isin R is the local amplitude and120601119896119897119898119899

isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860

119896119897119898119899

(12)

Since 119860119896119897119898119899

isin R we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot cos (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 119860119896119897119898119899

(13)

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 0

(14)

Note that119908(119909 minus 1198961198870 119910 minus 119897119887

0) sin(120596

119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) +

120601119896119897119898119899

) is a real-valued Gabor receptive field with a preferredphase at 120601

119896119897119898119899+ 1205872 minus 120596

119909119898

1198961198870minus 120596119910119899

1198971198870

Remark 3 Assuming that the local phase information 120601119896119897119898119899

is obtained via measurements that is filtering the image 119906

with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions

119908 (119909 minus 1198961198870 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

(15)

where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le

119879119909 0 le 119897119887

0le 119879119910 minusΩ119909le 119898120596

0le Ω119909 minusΩ119910le 1198991205960le Ω119910

We are now in the position to provide a reconstructionalgorithm of the image from phase 120601

119896119897119898119899 (119896 119897 119898 119899) isin I

Lemma 4 119906 can be reconstructed from 120601119896119897119898119899

(119896 119897 119898 119899) isin Ias

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (16)

with

Φ119888 = 0 (17)

whereΦ is a matrix whose 119901th row and 119902th column entry are

[Φ]119901119902

= int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910

(18)

Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+

119871119910+ 1) 119888 is a vector of the form

119888 = [119888minus119871119909minus119871119910

119888minus119871119909minus119871119910+1 119888

minus119871119909119871119910

119888minus119871119909+1minus119871

119910

119888minus119871119909+1minus119871

119910+1 119888

minus119871119909+1119871119910

119888119871119909minus119871119910

119888119871119909minus119871119910+1

119888119871119909119871119910

]

119879

(19)

that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871

119909+ 1)(2119871

119910+ 1) minus 1 where 119873 is the number of phase

measurements

Proof Substituting (7) into (14) we obtain119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910 = 0

(20)

for all 119896 119897 119898 119899 isin Z Therefore we have

Φ119888 = 0 (21)

inferring that 119888 is in the null space ofΦIf 119873 lt (2119871

119909+ 1)(2119871

119910+ 1) minus 1 it follows from the rank-

nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0

Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int

R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)

Computational Intelligence and Neuroscience 5

(a) (b) (c)

Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error

3 Visual Motion Detection fromPhase Information

In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field

31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0) (22)

where

119904119909(119905) = int

119905

0

V119909(119904) 119889119904

119904119910(119905) = int

119905

0

V119910(119904) 119889119904

(23)

are the total length of translation at time 119905 in each dimen-sion and V

119909(119905) and V

119910(119905) are the corresponding instanta-

neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following

Lemma 6 The change (derivative) of the global phase is givenby

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(24)

where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global

phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition

Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier

transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have

(F119906 (sdot sdot 119905)) (120596119909 120596119910)

= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596

119910119904119910(119905))

(25)

For a certain frequency component (120596119909 120596119910) the change of its

global phase over time amounts to

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909

119889119904119909(119905)

119889119905

minus 120596119910

119889119904119910(119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(26)

Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector

32 The Change of Local Phase

321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field

Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 4: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

4 Computational Intelligence and Neuroscience

where 119879119909= 2120587119871

119909Ω119909 119879119910= 2120587119871

119910Ω119910are the period of the

space in the 119909 and 119910 dimensions respectivelyConsider a bank of119873 Gabor receptive fields

ℎ119896119897119898119899

(119909 119910) = T119896119897(119908 (119909 119910) 119890

minus119895(120596119909119898119909+120596119910119899119910)) (10)

where T119896119897is the translation operator with T

119896119897119906 = 119906(119909 minus

1198961198870 119910 minus 119897119887

0) 119896 119897 isin Z 119887

0gt 0 0 le 119896119887

0le 119879119909 and 0 le

1198971198870le 119879119910 and 120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960 119898 119899 isin Z 120596

0gt 0

minusΩ119909le 119898120596

0le Ω119909 and minusΩ

119910le 1198991205960le Ω119910 The responses of

the Gabor receptive fields to the input 119906(119909 119910) are given by

int

R2119906 (119909 119910) ℎ

119896119897119898119899(119909 119910) 119889119909 119889119910 = int

R2119906 (119909 119910)

sdot 119908 (119909 minus 1198961198870 119910 minus 119897119887

0) 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870))119889119909 119889119910

= 119860119896119897119898119899

119890119895120601119896119897119898119899

(11)

where 119860119896119897119898119899

ge 0 119860119896119897119898119899

isin R is the local amplitude and120601119896119897119898119899

isin [0 2120587) is the local phaseDividing both sides of (11) by 119890119895120601119896119897119898119899 we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot 119890minus119895(120596119909119898(119909minus119896119887

0)+120596119910119899(119910minus1198971198870)+120601119896119897119898119899)119889119909 119889119910 = 119860

119896119897119898119899

(12)

Since 119860119896119897119898119899

isin R we have

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot cos (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 119860119896119897119898119899

(13)

int

R2119906 (119909 119910)119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

) 119889119909 119889119910

= 0

(14)

Note that119908(119909 minus 1198961198870 119910 minus 119897119887

0) sin(120596

119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) +

120601119896119897119898119899

) is a real-valued Gabor receptive field with a preferredphase at 120601

119896119897119898119899+ 1205872 minus 120596

119909119898

1198961198870minus 120596119910119899

1198971198870

Remark 3 Assuming that the local phase information 120601119896119897119898119899

is obtained via measurements that is filtering the image 119906

with pairs of Gabor receptive fields (10) the set of linearequations (14) has a simple interpretation the image 119906 isorthogonal to the space spanned by the functions

119908 (119909 minus 1198961198870 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

(15)

where (119896 119897 119898 119899) isin I with I = (119896 119897 119898 119899) isin Z4 | 0 le 1198961198870le

119879119909 0 le 119897119887

0le 119879119910 minusΩ119909le 119898120596

0le Ω119909 minusΩ119910le 1198991205960le Ω119910

We are now in the position to provide a reconstructionalgorithm of the image from phase 120601

119896119897119898119899 (119896 119897 119898 119899) isin I

Lemma 4 119906 can be reconstructed from 120601119896119897119898119899

(119896 119897 119898 119899) isin Ias

119906 (119909 119910) =

119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

119890119897119909119897119910

(119909 119910) (16)

with

Φ119888 = 0 (17)

whereΦ is a matrix whose 119901th row and 119902th column entry are

[Φ]119901119902

= int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910

(18)

Here 119901 traverses the set I and 119902 = (2119871119910+ 1)(119897119909+ 119871119909) + (119897119910+

119871119910+ 1) 119888 is a vector of the form

119888 = [119888minus119871119909minus119871119910

119888minus119871119909minus119871119910+1 119888

minus119871119909119871119910

119888minus119871119909+1minus119871

119910

119888minus119871119909+1minus119871

119910+1 119888

minus119871119909+1119871119910

119888119871119909minus119871119910

119888119871119909minus119871119910+1

119888119871119909119871119910

]

119879

(19)

that belongs to the null space of Φ A necessary conditionfor perfect reconstruction of 119906 up to a constant scale is that119873 ge (2119871

119909+ 1)(2119871

119910+ 1) minus 1 where 119873 is the number of phase

measurements

Proof Substituting (7) into (14) we obtain119871119909

sum

119897119909=minus119871119909

119871119910

sum

119897119910=minus119871119910

119888119897119909119897119910

int

R2119908 (119909 minus 119896119887

0 119910 minus 119897119887

0)

sdot sin (120596119909119898

(119909 minus 1198961198870) + 120596119910119899

(119910 minus 1198971198870) + 120601119896119897119898119899

)

sdot 119890119897119909119897119910

(119909 119910) 119889119909 119889119910 = 0

(20)

for all 119896 119897 119898 119899 isin Z Therefore we have

Φ119888 = 0 (21)

inferring that 119888 is in the null space ofΦIf 119873 lt (2119871

119909+ 1)(2119871

119910+ 1) minus 1 it follows from the rank-

nullity theorem that dim(null(Φ)) gt 1 leading to multiplelinearly independent solutions toΦ119888 = 0

Example 5 In Figure 2 an example of reconstruction of animage is shown using only local phase information Thereconstructed signal was scaled to match the original signalThe SNR of the reconstruction is 4448 [dB] An alternativeway to obtain a unique reconstruction is to include anadditional measurement for example the mean value of thesignal int

R2119906(119909 119910)119889119909 119889119910 to the system of linear equations (14)

Computational Intelligence and Neuroscience 5

(a) (b) (c)

Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error

3 Visual Motion Detection fromPhase Information

In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field

31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0) (22)

where

119904119909(119905) = int

119905

0

V119909(119904) 119889119904

119904119910(119905) = int

119905

0

V119910(119904) 119889119904

(23)

are the total length of translation at time 119905 in each dimen-sion and V

119909(119905) and V

119910(119905) are the corresponding instanta-

neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following

Lemma 6 The change (derivative) of the global phase is givenby

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(24)

where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global

phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition

Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier

transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have

(F119906 (sdot sdot 119905)) (120596119909 120596119910)

= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596

119910119904119910(119905))

(25)

For a certain frequency component (120596119909 120596119910) the change of its

global phase over time amounts to

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909

119889119904119909(119905)

119889119905

minus 120596119910

119889119904119910(119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(26)

Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector

32 The Change of Local Phase

321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field

Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 5: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 5

(a) (b) (c)

Figure 2 An example of reconstruction of image from local phase information (a) Original image the dimension of the space is 16129(b) Reconstructed image from 20102 phase measurements scaled to match the original (c) Error

3 Visual Motion Detection fromPhase Information

In this section we consider visual fields that change as afunction of time For notational simplicity 119906 will denote herethe space-time intensity of the visual field

31 The Global Phase Equation for Translational Motion Let119906 = 119906(119909 119910 119905) (119909 119910) isin R2 119905 isin R be a visual stimulus If thevisual stimulus is a pure translation of the signal at 119906(119909 119910 0)that is

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0) (22)

where

119904119909(119905) = int

119905

0

V119909(119904) 119889119904

119904119910(119905) = int

119905

0

V119910(119904) 119889119904

(23)

are the total length of translation at time 119905 in each dimen-sion and V

119909(119905) and V

119910(119905) are the corresponding instanta-

neous velocity components then the only difference between119906(119909 119910 119905) and 119906(119909 119910 0) in the Fourier domain is captured bytheir global phase More formally consider the following

Lemma 6 The change (derivative) of the global phase is givenby

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(24)

where by abuse of notation 120601(120596119909 120596119910 119905) denotes the global

phase of 119906(119909 119910 119905) and 120601(120596119909 120596119910 0) is the initial condition

Proof If (F119906(sdot sdot 0)) (120596119909 120596119910) is the 2D (spatial) Fourier

transform of 119906 = 119906(119909 119910 0) (119909 119910) isin R2 by the Fourier shifttheorem we have

(F119906 (sdot sdot 119905)) (120596119909 120596119910)

= (F119906 (sdot sdot 0)) (120596119909 120596119910) 119890minus119895(120596119909119904119909(119905)+120596

119910119904119910(119905))

(25)

For a certain frequency component (120596119909 120596119910) the change of its

global phase over time amounts to

119889120601 (120596119909 120596119910 119905)

119889119905

= minus120596119909

119889119904119909(119905)

119889119905

minus 120596119910

119889119904119910(119905)

119889119905

= minus120596119909V119909(119905) minus 120596

119910V119910(119905)

= minus [120596119909 120596119910] [V119909(119905) V119910(119905)]

119879

(26)

Therefore in the simple case where the entire visual field isshifting the derivative of the phase of Fourier componentsindicates motion and it can be obtained by the innerproduct between the component frequency and the velocityvector

32 The Change of Local Phase

321The Local Phase Equation for Translational Motion Theanalysis in Section 31 applies to global motion This type ofmotion occurs most frequently when the imaging deviceeither an eye or a cameramoves Visualmotion in the naturalenvironment however ismore diverse across the screen sinceit is often time produced by multiple moving objects Theobjects can be small and the motion more localized in thevisual field

Taking the global phase of 119906 will not simply revealwhere motion of independent objects takes place or theirdirectionvelocity of motion The ease of interpretation ofmotion by using the Fourier transform however motivates

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 6: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

6 Computational Intelligence and Neuroscience

us to reuse the same concept in detecting local motion Thiscan be achieved by restricting the domain of the visual fieldwhere the Fourier transform is applied

To be able to detect local motion we consider thelocal phase of 119906(119909 119910 119905) by taking the STFT with windowfunction 119908(119909 119910) Note that the STFT and its ubiquitousimplementation in DSP chips can be extensively used in anydimension For simplicity and without loss of generality weconsider the window to be centered at (0 0) The STFT isgiven by

int

R2119906 (119909 119910 119905) 119908 (119909 119910) 119890

minus119895(120596119909119909+120596119910119910)119889119909 119889119910

= 11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

(27)

where by abuse of notation 11986000(120596119909 120596119910 119905) is the amplitude

and 12060100(120596119909 120596119910 119905) the local phase

Before we move on to the mathematical analysis we canintuitively explain the relation between the change in localphase and visual motion taking place across the windowsupport First if the stimulus undergoes a uniform changeof intensity or it changes proportionally over time due tolighting conditions for example the local phase does notchange since the phase is invariant with respect to intensityscaling Therefore the local phase does not change for suchnonmotion stimuli Second a rigid edge moving across thewindow support will induce a phase change

For a strictly translational signal within the windowsupport (footprint) for example

119906 (119909 119910 119905) = 119906 (119909 minus 119904119909(119905) 119910 minus 119904

119910(119905) 0)

for (119909 119910) isin supp (119908 (119909 119910))

(28)

where 119904119909(119905) and 119904

119910(119905) are as defined in (23) we have the

following result

Lemma 7 Consider the following

11988912060100

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909(119905)

119889119905

120596119909minus

119889119904119910(119905)

119889119905

120596119910

+ v00(120596119909 120596119910 119905)

(29)

where by abuse of notation 12060100(120596119909 120596119910 119905) is the local phase

of 119906(119909 119910 119905) and 12060100(120596119909 120596119910 0) is the initial condition The

functional form of the term v00

= v00(120596119909 120596119910 119905) is provided

in the Appendix

Proof The derivation of (29) and the functional form of v00

are given in the Appendix

Remark 8 Wenotice that the derivative of the local phase hassimilar structure to that of the global phase but for the addedterm v

00Through simulations we observed that the first two

terms in (29) dominate over the last term for an ON or OFFmoving edge [3] For example Figure 3 shows the derivativeof the local phase given in (29) for an ON edge moving withvelocity (40 0) pixelssec

42

0minus2

minus4

100

d120601dt

120596yminus250minus200minus150minus100minus50

050

150

200250

31 2minus4 0minus1minus3 minus2 4

120596x

Figure 3 Derivative of the local phase for an ON edge moving inthe positive 119909 direction

Remark 9 Note that 12060100(120596119909 120596119910 119905) may not be differen-

tiable even if 119906(119909 119910 119905) is differentiable particularly when11986000(120596119909 120596119910 119905) = 0 For example the spatial phase can jump

from a positive value to zero when11986000(120596119909 120596119910 119905) diminishes

This also suggests that the instantaneous local spatial phaseis less informative about a region of a visual scene whenever11986000(120596119909 120596119910 119905) is close to zero Nevertheless the time deriva-

tive of the local phase can be approximated by applying ahigh-pass filter to 120601

00(120596119909 120596119910 119905)

322 The Block Structure for Computing the Local Phase Weconstruct Gaussian windows along the 119909 119910 dimensions TheGaussian windows are defined as

(T119896119897119908) (119909 119910) = 119890

minus((119909minus119909119896)2+(119909minus119910

119897)2)21205902

(30)

where 119909119896= 1198961198870 119910119897= 1198971198870 in which 119887

0isin Z+ is the distance

between two neighboring windows and 1 le 1198961198870le 119875119909 1 le

1198971198870le 119875119910 where 119875

119909 119875119910isin Z+ are the number of pixels of the

screen in 119909 and 119910 directions respectivelyWe then take the 2D Fourier transform of the windowed

video signal 119906(119909 119910 119905)(T119896119897119908)(119909 119910) and write in polar form

int

R2119906 (119909 119910 119905) (T

119896119897119908) (119909 119910) 119890

minus119895(120596119909(119909minus119909119896)+120596119910(119910minus119910119897))119889119909 119889119910

= 119860119896119897(120596119909 120596119910 119905) 119890119895120601119896119897(120596119909120596119910119905)

(31)

The above integral can be very efficiently evaluated usingthe 2D FFT in discrete domain defined on 119872 times 119872 blocksapproximating the footprint of the Gaussian windows Forexample the standard deviation of the Gaussian windows weuse in the examples in Section 4 is 4 pixels A block of 32times32

pixels (119872 = 32) is sufficient to cover the effective support(or footprint) of the Gaussian window At the same time thesize of the block is a power of 2 which is most suitable forFFT-based computationThe processing of each block (119896 119897) isindependent of all other blocks thereby parallelism is readilyachieved

Note that the size of the window is informed by thesize of the objects one is interested in locating Measure-ments of the local phase using smaller window functions

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 7: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 7

60

50

40

30

20

10

y (pi

xel)

01

02

03

04

05

06

07

08

09

1

20 30 40 50 6010x (pixel)

(a)

0

02

04

06

08

1

Mag

nitu

de (a

u)

10 20 30 40 50 60 700y (pixel)

(b)

Figure 4 Example of block structure (a) Four yellow disks show 4 Gaussian window functions with translation parameter 119896 = 3 119897 = 3

(top-left) 119896 = 3 119897 = 7 (bottom-left) 119896 = 7 119897 = 3 (top-right) and 119896 = 7 119897 = 7 (bottom-right) The red solid square shows a 32 times 32-pixel blockapproximating the Gaussian window with 119896 = 3 119897 = 3 and the green dashed square shows a 32 times 32-pixel block approximating the Gaussianwindow with 119896 = 3 119897 = 7 (b) Cross section of all 11 Gaussian windows with translation parameters 119896 = 7 119897 isin [0 10] The cross section istaken as indicated by the magenta line in (a) and the red and blue curve correspond to cross sections of the two Gaussian windows shown in(a) centered on the magenta line

are less robust to noise Larger windows would enhanceobject motion detection if the object size is comparable tothe window size However there would be an increasedlikelihood of independent movement of multiple objectswithin the same window which is not modeled here andthereby may not be robustly detected

Therefore for each block (119896 119897) we obtain 1198722 measure-

ments of the phase 120601119896119897(120596119909119898

120596119910119898

119905) at every time instant 119905with (120596

119909119898

120596119910119898

) isin D2 where

D2= (120596

119909119898

= 1198981205960 120596119910119899

= 1198991205960) 119898 119899 = minus1198722

minus 1198722 + 1 1198722 minus 1

(32)

with 1205960= 2120587119872 We then compute the temporal derivative

of the phase that is (119889120601119896119897119889119905)(120596

119909 120596119910 119905) for (120596

119909 120596119910) isin D2

We further illustrate an example of the block structurein Figure 4 Figure 4(a) shows an example of an image of64 times 64 pixels Four Gaussian windows are shown each witha standard deviation of 4 pixels The distance between thecenters of two neighboring Gaussian windows is 6 pixelsThe red solid square shows a 32 times 32-pixel block with 119896 =

3 119897 = 3 which encloses effective support of the Gaussianwindow on top-left (119896 = 0 119897 = 0 is the block with Gaussianwindow centered at pixel (1 1)) The green dashed squareshows another 32 times 32-pixel block with 119896 = 3 119897 = 7 Thetwo Gaussian windows on the right are associated with theblocks 119896 = 7 119897 = 3 and 119896 = 7 119897 = 7 respectively Crosssection of all Gaussian windows with 119896 = 7 119897 isin [0 10] that isthose centered on themagenta line are shown in Figure 4(b)The red and blue curve in Figure 4(b) correspond to the two

Gaussian windows shown in Figure 4(a) Figure 4(b) alsosuggests that some of the Gaussian windows are cut off on theboundaries This is however equivalent to assuming that thepixel values outside the boundary are always zero and it willnot significantly affect motion detection based on the changeof local phase

Since the phase and thereby the phase change is noisierwhen the local amplitude is low an additional denoisingstep can be employed to discount the measurements of(119889120601119896119897119889119905)(120596

119909 120596119910 119905) for low amplitude values 119860

119896119897(120596119909 120596119910 119905)

The denoising is given by

119889120601119896119897

119889119905

(120596119909 120596119910 119905)

sdot

119860119896119897(120596119909 120596119910 119905)

(11198722)sum(120596119909120596119910)isinD2 119860119896119897 (120596119909 120596119910 119905) + 120598

(33)

where 120598 gt 0 is a constant and (120596119909 120596119910) isin D2

33 The Phase-Based Detector We propose here a block FFTbased algorithm to detect motion using phase informationSuch an algorithm is due to its simplicity and parallelismhighly suitable for an in silico implementation

331 Radon Transform on the Change of Phases We exploitthe approximately linear structure of the phase derivative forblocks exhibiting motion by computing the Radon transformof (119889120601

119896119897119889119905)(120596

119909 120596119910 119905) over a circular bounded domain 119862 =

(120596119909 120596119910) | (120596119909 120596119910) isin D2 1205962

119909+ 1205962

119910lt 1205872

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 8: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

8 Computational Intelligence and Neuroscience

The Radon transform of the change of phase in thedomain 119862 is given by

(R119889120601119896119897

119889119905

) (120588 120579 119905) = int

R

119889120601119896119897

119889119905

(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588

sdot sin 120579 + 119904 sdot cos 120579 119905) 1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579

+ 119904 sdot cos 120579) 119889119904

(34)

where

1119862(120596119909 120596119910) =

1 if (120596119909 120596119910) isin 119862

0 otherwise(35)

The Radon transform (R(119889120601119896119897119889119905))(120588 120579 119905) evaluated at

a particular point (1205880 1205790 1199050) is essentially an integral

of (119889120601119896119897119889119905)(120596

119909 120596119910 1199050) along a line oriented at angle

1205872 + 1205790with the 120596

119909axis and at distance |120588

0| along the

(cos(1205790) sin(120579

0)) direction from (0 0)

If for a particular 119896 and 119897 we have (119889120601119896119897119889119905)(120596

119909 120596119910 119905) =

minusV119909(119905)120596119909minus V119910(119905)120596119910 we have

(R (119889120601119896119897119889119905)) (120588 120579 119905)

c (120588 120579)

= 120588 [minusV119909(119905) cos 120579 minus V

119910(119905) sin 120579]

(36)

where

c (120588 120579)

= int

R

1119862(120588 sdot cos 120579 minus 119904 sdot sin 120579 120588 sdot sin 120579 + 119904 sdot cos 120579) 119889119904

(37)

c(120588 120579) is a correction term due to different length of lineintegrals for different values of (120588 120579) in the bounded domain119862

After computing the Radon transform of (119889120601119896119897119889119905)(120596

119909

120596119910 119905) for every block (119896 119897) at time 119905

0 we compute the Phase

Motion Indicator (PMI) defined as

PMI119896119897= max120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(38)

If the PMI119896119897

is larger than a chosen threshold motion isdeemed to occur in block (119896 119897) at time 119905

0

Using the Radon transform makes it easier to separaterigid motion from noise Since the phase is quite sensitiveto noise particularly when the amplitude is very smallthe change of phase under noise may have comparablemagnitude to that due to motion as mentioned earlier Thechange of phase under noise however does not possess thestructure suggested by (29) in the (120596

119909 120596119910) domain Instead

it appears to be more randomly distributed Consequently

the PMI value is comparatively small for these blocks (see alsoSection 333)

Moreover the direction of motion for block (119896 119897) wheremotion is detected can be easily computed as

120579119896119897

= 120587(

sign (sum120588gt0

(R (119889120601119896119897119889119905)) (120588 120572

119896119897 1199050) c (120588 120572

119896119897)) + 1

2

)

+ 120572119896119897

(39)

where

120572119896119897= argmax120579isin[0120587)

sum

120588

100381610038161003816100381610038161003816100381610038161003816

(R (119889120601119896119897119889119905)) (120588 120579 119905

0)

c (120588 120579)

100381610038161003816100381610038161003816100381610038161003816

(40)

This follows from (36)

332 The Phase-Based Motion Detection Algorithm We for-mally summarize the above analysis as Algorithm 1 Figure 5shows a schematic diagram of the proposed phase-basedmotion detection algorithm

The algorithm is subdivided into two parts The first partcomputes local phase changes and the second part is thephase-based motion detector

In the first part the screen is divided into overlappingblocks For example the red green and blue blocks in theplane ldquodivide into overlapping blocksrdquo correspond to thesquares of the same color covering the video stream AGaussian window is then applied on each block followed bya 2D FFT operation that is used to extract the local phase Atemporal high-pass filter is then employed to extract phasechanges

In the second part the PMI is evaluated for each blockbased on the Radon transform of the local phase changes ineach block Motion is detected for blocks with PMI largerthan a preset threshold and the direction of motion iscomputed as in (39)

It is easy to notice that the algorithm can be highlyparallelized

333 Example We provide an illustrative example inFigure 6 showing how motion is detected using Algorithm 1The full video of this example can be found in SupplementaryVideo S1 see Supplementary Material available online athttpdxdoiorg10115520167915245 Figure 6(a) depicts astill from the ldquohighway videordquo in the Change Detection 2014dataset [22] evaluated at a particular time 119905

0 As suggested

by the algorithm the screen in Figure 6(a) is divided into26 times 19 overlapping blocks and the window functions areapplied to each block Local phases can then be extractedfrom the 2DFFT of eachwindowed block and the local phasechanges are obtained by temporal high-pass filtering Thephase change is shown in Figure 6(b) for all blocks with block(12 11) enlarged in Figure 6(c) and block (23 6) enlarged inFigure 6(d) (see also the plane ldquo2D FFT and extract phasechangerdquo in Figure 5) Note that at the time of the videoframe block (12 11) covers a part of the vehicle in motion

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 9: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 9

Input Visual stream or Video data 119906(119909 119910 119905)Output Detected MotionConstruct Gaussian window of size119872times119872 denote as 119908for 119905 = Δ119879 2Δ119879 119873Δ119879 do

Divide the screen of frame at time 119905 into multiple119872times119872 blocks 119906119896119897(119905) with overlaps

foreach block 119906119896119897(119905) do

Multiply pixel-wise the block 119906119896119897(119905) with the Gaussian window 119906

119896119897(119905)119908

Take119872times119872 2D FFT of 119906119896119897(119905)119908 denote phase as 120601

119896119897(120596119909 120596119910 119905) and amplitude as 119860

119896119897(120596119909 120596119910 119905)

foreach frequency (120596119909 120596119910) do

Highpass filter temporally 120601119896119897(120596119909 120596119910 119905) denote 1206011015840

119896119897(120596119909 120596119910 119905)

Optionally denoise by 1206011015840119896119897(120596119909 120596119910 119905) larr 120601

1015840

119896119897(120596119909 120596119910 119905)(119860

119896119897(120596119909 120596119910 119905)((1119872

2) sum(120596119909 120596119910)isinD

2 119860119896119897(120596119909 120596119910 119905) + 120598))

endCompute the radon transform (R120601

1015840

119896119897)(120588 120579 119905)

Compute PMI119896119897(119905) according to (38)

if PMI119896119897(119905) gt threshold then

Motion Detected at block (119896 119897) at time 119905Compute direction of motion according to (39)

endend

end

Algorithm 1 Phase-based motion detection algorithm using the FFT

y

120596x

120596y

y

t

120596x

120596y

x

y

x

y

x

x

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

middot middot middot

Divided intooverlapping blocks

Apply window function

2D FFT andextract phase change

Radon transformand extract strength

and orientation of plane

Thresholding

Computephase

change

The phase-baseddetector

Figure 5 Schematic diagram of proposed phase-based motion detection algorithm

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 10: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

10 Computational Intelligence and Neuroscience

in the front and block (23 6) corresponds to an area of thehighway pavement where no motion occurs

Figure 6(f) depicts for each block (119896 119897) the maximumphase change over all (120596

119909 120596119910) isin D2 that is

max(120596119909120596119910)isinD2

10038161003816100381610038161003816100381610038161003816

119889120601119896119897

119889119905

(120596119909 120596119910 1199050)

10038161003816100381610038161003816100381610038161003816

(41)

We observe from the figure that for regions with lowamplitude such as the region depicting the road whenthe normalization constant is absent the derivative of thephase can be noisy For these blocks the maximum of|(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)| over all (120596

119909 120596119910) isin D2 is comparable

to the maximum obtained for blocks that cover the vehiclesin motion

However (29) suggests that the local phase change frommultiple filter pairs centered at the same spatial position(119896 119897) can provide a constraint to robustly estimate motionand its direction Given the block structure employed in thecomputation of the local phase it is natural to utilize phasechange information from multiple sources

Indeed if for a particular block (119896 119897) (119889120601119896119897119889119905)(120596

119909

120596119910 119905) = minusV

119909(119905)120596119909minus V119910(119905)120596119910 then it is easy to see that (119889120601

119896119897

119889119905)(120596119909 120596119910 119905) will be zero on the line V

119909(119905)120596119909+ V119910(119905)120596119910= 0

and have opposite sign on either side of this line For examplein Figures 6(b) and 6(c) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) clearly exhibits

this property for blocks that cover a vehicle in motion ThePMI is a tool to evaluate this property

Finally the PMIs for all blocks are shown compactlyin a heat map in Figure 6(e) The figure shows clearly thatthe blocks corresponding to the two moving vehicles have ahigh PMI value while the stationary background areas havea low PMI value allowing one to easily detect motion byemploying simple thresholding (see also the plane ldquoRadontransform and extract strength orientation of planerdquo inFigure 5) In addition the orientation ofmotion in each blockis readily observable even by inspection in Figure 6(b) by aline separating the yellow part and blue part in each blockFurther results about the direction of motion are presentedin Section 4

34 Relationship to Biological Motion Detectors A straight-forward way to implement local motion detectors is to applya complex-valued Gabor receptive field (5) to the video signal119906 and then take the derivative of the phase with respect totime or apply a high-pass filter on the phase to approximatethe derivative

We present here an alternate implementation withoutexplicitly computing the phaseThiswill elucidate the relationbetween the phase-based motion detector presented in theprevious section and some elementary motion detectionmodels used in biology such as theReichardtmotion detector[7] and motion energy detector [9]

Assuming that the local phase 12060100(120596119909 120596119910 119905) is differen-

tiable we have (see also the Appendix)

11988912060100(120596119909 120596119910 119905)

119889119905

=

(119889119887 (120596119909 120596119910 119905) 119889119905) 119886 (120596

119909 120596119910 119905) minus (119889119886 (120596

119909 120596119910 119905) 119889119905) 119887 (120596

119909 120596119910 119905)

[119886 (120596119909 120596119910 119905)]

2

+ [119887 (120596119909 120596119910 119905)]

2 (42)

where 119886(120596119909 120596119910 119905) and 119887(120596

119909 120596119910 119905) are respectively the real

and imaginary parts of 11986000(120596119909 120596119910 119905)11989011989512060100(120596119909120596119910119905)

We notice that the denominator of (42) is the square ofthe local amplitude of 119906 and the numerator is of the formof a second order Volterra kernel This suggests that thetime derivative of the local phase can be viewed as a secondorder Volterra kernel that processes two normalized spatiallyfiltered inputs V

1and V2

We consider an elaborated Reichardt motion detector asshown in Figure 7 It is equipped with a quadrature pair ofGabor filters whose outputs are 119903

1(119905) = 119886(120596

119909 120596119910 119905) and

1199032(119905) = 119887(120596

119909 120596119910 119905) respectively for a particular value of

(120596119909 120596119910) The pair of Gabor filters that provide these outputs

are the real and imaginary parts of119908(119909 119910)119890minus119895(120596119909119909+120596119910119910) It also

consists of a temporal high-pass 1198921(119905) and temporal low-

pass filter 1198922(119905) [9] The output of the elaborated Reichardt

detector follows the diagram in Figure 7 and can be expressedas

(1199032lowast 1198921) (119905) (119903

1lowast 1198922) (119905) minus (119903

1lowast 1198921) (119905) (119903

2lowast 1198922) (119905) (43)

The response can also be characterized by a second orderVolterra kernel We notice the striking similarity between

(43) and the numerator of (42) In fact the phase-basedmotion detector shares some properties with the Reichardtmotion detector For example it is straightforward to see thata single phase-basedmotion detector is tuned to the temporalfrequency of a moving sinusoidal grating

Since the motion energy detector is formally equivalentto an elaborated Reichardt motion detector [9] the structureof the motion energy detector with divisive normalization isalso similar to the phase-based motion detector

4 Exploratory Results

In this section we apply the phase-based motion detectionalgorithm on several video sequences and demonstrate itsefficiency and effectiveness in detecting local motion Themotion detection algorithm is compared to two well-knownbiological motion detectors namely the Reichardt motiondetector and the Barlow-Levick motion detector We thenshow that the detected local motion can be used in motionsegmentation tasks The effectiveness of the segmentationis compared to segmentation using motion informationobtained from a widely used optic flow based algorithmavailable in the literature [4]

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 11: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 11

(f) (e)

(b)

(a)

(d)

(c)

minus120587 minus1205872 0 1205872

10080604020

y (pi

xel)

100 15050x (pixel)

0020406081

15

10

5

l

01234567+

15

10

5

l

0510152025+

minus6

minus4

minus2

0

2

4

6

181716151413121110

9876543210

l

2 4 6 8 10 12 14 16 18 20 22 240k

minus6minus4minus20246

minus6minus4minus20246

120596x

minus120587 minus1205872 0 1205872

120596x

1205872

0

minus1205872

minus120587

120596y

1205872

0

minus1205872

minus120587

120596y

10 15 20 255k

10 20 255 15k

Figure 6 Evaluating the motion detection algorithm at time 1199050(see also Supplementary Video S1) (a) A still from a 156 times 112-pixel video at

time 1199050 (b) (119889120601

119896119897119889119905)(120596

119909 120596119910 1199050) for all 26times19 blocks each of size 32times32The blocks are concatenated to create a large ldquophase imagerdquo with their

relative neighbors kept (c) (1198891206011211

119889119905)(120596119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2 space (d) (119889120601

236119889119905)(120596

119909 120596119910 1199050) shown in the (120596

119909 120596119910) isin D2

space (e) Phase Motion Indicator (PMI) for all blocks at 1199050 Each pixel (119896 119897) is color coded for PMI

119896119897 (f) max

(120596119909 120596119910)isinD2 |(119889120601119896119897119889119905)(120596

119909 120596119910 1199050)|

for all blocks at 1199050 Each pixel (119896 119897) is color coded for this maximum value of block (119896 119897)

g1(t)g2(t) g1(t) g2(t)

times times

minus++

SpatialGaborfilter

SpatialGaborfilter

r1(t) r2(t)

u(x y t)

Figure 7 Diagram of an elaborated Reichardt motion detector Itconsists of a quadrature pair of Gabor filters whose outputs 119903

1and 1199032

provide inputs to the high-pass filters 1198921(119905) and the low-pass filters

1198922(119905)

Table 1 Processing Speeds of the proposed algorithm the Reichardtmotion detector and the Barlow-Levick Motion detector

Screen size Processing capability (frames per second)Proposed Reichardt Barlow-Levick

320 times 240 420 790 800720 times 576 135 685 6901280 times 720 60 274 2931920 times 1080 27 191 194

41 Efficient Parallel Implementation The algorithm wasimplemented in PyCUDA [23] and tested on an NVIDIAGeForce GTX TITAN GPU All computations use singleprecision floating points The processing speeds of the algo-rithm for several screen sizes are listed in Table 1 Clearlythe proposed phase-based motion detection algorithm hasreal-time capability to process video even with full HighDefinition screen size

For comparison we implemented the Reichardt motiondetector and the Barlow-Levick motion detector Theirrespective diagrams are shown in Figure 8 Note that we

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 12: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

12 Computational Intelligence and Neuroscience

ukl(t) uk+1l(t)

g1(t)

g2(t)

g1(t)

g2(t)

+ minus+

timestimes

(a)

minus

minus minus

+

ukl(t) uk+1l(t)

g1(t)g1(t)

g2(t) g2(t)

++

+

(b)

Figure 8 (a) A Reichardt motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video

(b) A Barlow-Levick motion detector detecting motion between the two points 119906119896119897and 119906

119896+1119897in the blurred and downsampled video 119892

1(119905)

denotes a high-pass filter and 1198922(119905) denotes a low-pass filter

moved the high-pass filter ℎ1(119905) to the front of the low-pass fil-

ter ℎ2(119905) This configuration provides a superior performance

to the one in Figure 7 For both the Reichardt and the Barlow-Levick detectors the videos are first blurred by a Gaussianfilter (with the same variance as the Gaussian window in (30)used for the phase-based motion detector) and subsampledat the center of each overlapping block in the phase-basedmotion detector The subsampled video then provides inputsto two 2D arrays of the circuits shown in Figure 8 one forthe horizontal direction and one for the vertical directionThe outputs of the horizontal and vertical motion circuitsform an array of motion vectors that indicate the strengthand direction of motion The three tested motion detectorshave the same number of outputs as a result The processingspeeds of the Reichardt motion detector and the Barlow-Levick motion detector both implemented in PyCUDA andtested on the same GPU are shown in Table 1

Note that the Reichardt motion detector and the Barlow-Levick motion detector are highly efficient due to the sim-plicity of their algorithmsThe phase-basedmotion detectionalgorithm however is a much more sophisticated algorithmand yet it can be implemented in real-time using parallelcomputing devices

The fast GPU implementation is based on the FFTand Matrix-Matrix multiplication It is expected that thoseoperations can be efficiently implemented in hardware forexample FPGA

42 Examples of Phased-BasedMotionDetection Weappliedour motion detection algorithm on video sequences of theChange Detection 2014 dataset [22] that did not exhibitcamera egomotion For these video sequences the standarddeviation of the Gaussian window functions was set to 4

pixels and the block size was chosen to be 32 times 32 pixelsThreshold and normalization parameters were kept the samewith the exception of the ldquothermal videordquo (in order to dealwith larger background noise levels see below) We alsotested the same video sequences using the Reichardt motiondetector and the Barlow-Levick motion detector For theReichardt detector the high-pass filters were chosen as firstorder filters with a time constant of 200 milliseconds and thelow-pass filters were chosen as first order filters with a timeconstant of 300 milliseconds (assuming that the frame rate is50 frames per second)Thresholdwas set to 2 For the Barlow-Levick motion detector the time constant of the first high-pass filters was set to 250 milliseconds The low-pass filterswere the same as in the Reichardt detectorThreshold was setto 2

The first video was taken from a highway surveillancecamera (ldquohighway videordquo) under good illumination con-ditions and high contrast The video had moderate noiseparticularly on the road surface The detected motion isshown in the top left panel of Figure 9 (see SupplementaryVideo S2 for full video) The phase-based motion detectionalgorithm captured both the moving cars and the tree leavesmoving (due to the wind) In the time interval between the9th and 10th second the camera was slightly moved left-and right-wards within 5 frames again possibly due to thewind Movement due to this shift was captured by themotiondetection algorithm and the algorithm was fast enough tocorrectly determine the direction of this movement In thisvideo we already noted that this algorithm suffers from theaperture problem For example in front of the van where along horizontal edge is present the detectedmotion ismostlypointing downwards In addition to moving downwards theedge is alsomoving to the left howeverThis is expected since

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 13: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 13

(a) (b) (c)

Figure 9 Top motion detection of the ldquohighway videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) Red arrows indicate detected motion and its direction Bottom the contrast of the video was artificiallyreduced by 5-fold and the mean was reduced to 35 of the original The red arrows are duplicated from the motion detection result on theoriginal video as in top Blue arrows are the result of motion detection on the video with reduced contrast If motion is detected both in theoriginal video and in the video with reduced contrast then the arrow is shown in magenta (as a mix of blue and red) (see SupplementaryVideo S2 for full video)

the algorithm only detects motion locally and does not takeinto account the overall shape of any object

For comparison the motion detection results for theReichardt motion detector and the Barlow-Levick motiondetector are shown in the top middle and top right panelof Figure 9 (see Supplementary Video S2 for full video) TheReichardt detector performed relatively well when vehiclesare moving faster but the direction it predicts for vehiclesmoving slower for example on the back of the image is notaccurate In addition motion is still detected in some partsof the screen where the vehicles have just passed by Thedetection result for the Barlow-Levick motion detector waspoorer In particular the response to OFF edge movement isalways the opposite to the actual movement direction

We then squeezed the range of the screen intensity from[0 1] to [02 04] resulting in a video with a lower meanluminance and lower contrast The motion detection resultson the low-contrast video are shown in the bottom 3 panelsof Figure 9 (see Supplementary Video S2 for full video) Forreference the motion detection results for the original videoare shown in red arrows Motion detected in the low-contrastvideo is shown in blue arrows if no motion is detected forthe block in the original video If motion is detected in boththe original and the low-contrast video the arrows are shownin magenta Figure 9 clearly shows that while the motiondetection performance is degraded for all three detectors thephase-basedmotion detector performed still quite well in thelow-contrast video detecting most of the moving vehicles

The other two detectors missed many of the blocks wheremotion was detected in the original movie

To quantify how well each detector works under differentcontrast conditions we computed the ratio of unthresholdedoutput values of eachmotion detector between lower contrastand full contrast video For example in the case of phase-basedmotion detector we computed for each block the ratiobetween the PMI index for the lower contrast video and thatfor the full contrast videoThe ratios are then averaged acrossall blocks where motion is detected in the full contrast videoThis average for different contrasts is shown in Figure 10 bythe blue curve In the ideal case when the normalizationconstant 120598 is 0 the phase detector should produce invariantresponses to videos with different contrast The curve shownhere is mainly due to a nonzero 120598The ratios for the Reichardtmotion detector and the Barlow-Levick motion detectorare computed similarly and are shown in red and yellowrespectively It is clear that the phase-based motion detectorhas a superior performance across a range of contrast valuesAt 20 contrast the phase-based detector still has 50 ofthe PMI index value for full contrast video As expectedthe response of Barlow-Levick motion detector is linear withrespect to contrast and the Reichardt motion detector has aquadratic relation to contrast For a fixed threshold value alarger ratio equates to amore consistent performance at lowercontrast

The second video was captured by a surveillance camerain a train station (ldquotrain station videordquo) The video was

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 14: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

14 Computational Intelligence and Neuroscience

Phase-basedReichardtBarlow-Levick

0

02

04

06

08

1

Ratio

40 60 80200 100Contrast ()

Figure 10 The ratio for the phase-based motion detector is definedas follows we set the motion detected in the 100 contrast case asthe baseline we compute the ratio between the PMI index for thelower contrast video and that for the full contrast video for eachblock the ratios are averaged across all the blocks where motion isdetected in the full contrast video the average is given as the ratio forthe phase-basedmotion detectorThe ratio for the Reichardtmotiondetector and the Barlow-Levickmotion detector is similarly defined

under moderate room light with a low noise level The frontside of the video had high contrast illumination on theback side was quite low The detected motion is shown inthe video of Figure 11 (see Supplementary Video S3 for fullvideo) Movements of people were successfully captured bythe motion detection algorithm

The third video was a ldquothermal videordquo with large amountof background noise (thermal video) The threshold fordetecting motion was raised by 60 in order to mitigate theincreased level of noise The detected motion is shown inthe video of Figure 12 (see Supplementary Video S4 for fullvideo)

The last example we show here was taken from a highwaysurveillance camera at night (ldquowinterstreet videordquo) Theoverall illumination on the lower-left side was low whereasillumination was moderate on the upper-right side wherethe road was covered by snow The detected motions areshown in the video in Figure 13 (see Supplementary VideoS5 for full video) We note that overall car movements weresuccessfully detected Car movements on the lower-left sidehowever suffered from low illumination and some parts ofthe car were not detected well due to the trade-off employedfor noise suppression

With a higher threshold the phase-basemotion detectionalgorithm is able to detect motion under noisy conditionsWe added to the original ldquohighway videordquo and ldquotrain stationvideordquo Gaussian white noise with standard deviation 5of the maximum luminance range The results are shownrespectively in Figures 14 and 15 (see Supplementary VideosS6 and S7 for full videos)

43 Examples of Motion Segmentation We asked whetherthe detected motion signals in the video sequences can beuseful for segmenting moving objects from the backgroundWe applied a larger threshold to only signalmotion for salient

objects The 32 times 32 blocks however introduce large bound-aries around themoving objects To reduce the boundary andto segment themoving objectmore closely to the actual objectboundary we applied themotion detection algorithm aroundthe detected boundary with 16 times 16 blocks If 16 times 16 blocksdid not indicate motion then the corresponding area wasremoved from the segmented object area

For comparison we performed motion segmentationbased on local motion detection based on an optic flow algo-rithm [4] The segmentation in this case was implementedby comparing the length of the optic flow vectors with anappropriate threshold

We employed a simple thresholding for both phase-basedmotion detection algorithms and optic flow based motiondetection More sophisticated algorithmsmay produce bettersegmentation results Thus the segmentation was purelybased onmotion cues and no postprocessing at pixel level wasperformedThe state-of-the-art results for the Change Detec-tion 2014 dataset utilize multiple cues such as motion colorand background extraction to segment objects and therebyachieve better results We are exploring here however onlythe case where motion is the only cue for segmentationTherefore the ground truth information from the dataset wasnot applicable to our test We will therefore only show theeffectiveness of motion segmentation visually

We first applied motion based segmentation on 2-secondsegment of the ldquohighway videordquo The result using the localphase-based motion detection algorithm is shown in thevideo of Figure 16(a) (see Supplementary Video S8a for fullvideo) and that using optic flow based motion detectionalgorithm is shown in the video of Figure 16(b) (see Supple-mentary Video S8b for full video) Both videos are playedback at 14 of speed With a higher threshold the movementof the leaves was no longer picked up by the phase-basedmotion detector Therefore only the cars were identifiedas moving objects and they are indicated in red Althoughthe moving objects were not perfectly segmented on theirboundary they were mostly captured For the optic flowbased segmentation since the regions of interest are set bythresholding the length of the velocity vector objects movingat lower speed for example the cars on the top were notalways picked up by the segmentation algorithm

We then applied the motion segmentation to a 2-secondsegment of the ldquotrain station videordquo the ldquothermal videordquo andthe ldquowinterstreet videordquo The results are shown respectivelyin the videos of Figures 17 18 and 19 (see SupplementaryVideos S9a and S9b S10a and S10b and S11a and S11b respfor full videos)

These results show that for the purpose of detecting localmotion and its use as a motion segmentation cue the localphase-basedmotion detectorworks as good if not better thana simple thresholding segmentation using an optic flow basedalgorithm

5 Discussion

Previous research demonstrated that global phase informa-tion alone can be used to faithfully represent visual scenesHere we provided a reconstruction algorithm of visual scenes

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 15: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 15

(a) (b) (c)

Figure 11 Motion detection of the ldquotrain station videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S3 for full video)

(a) (b) (c)

Figure 12 Motion detection of the ldquothermal videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and the Barlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S4 for full video)

by only using local phase information More importantlylocal phase information can be effectively used to detect localmotion Through a simple temporal derivative of the phasewe obtained a second order Volterra kernel that is appliedon two normalized inputs The structure of the second orderVolterra kernel in the phase-based motion detector is akin to

models employed to detect motion in biological vision forexample the Reichardt detector [7] and the motion energydetector [9]

We then proposed an efficient FFT-based algorithmemploying the change in local phase for detecting motionIn order to exploit the special structure of the change in

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 16: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

16 Computational Intelligence and Neuroscience

(a) (b) (c)

Figure 13 Motion detection of the ldquowinterstreet videordquo using the phase-based algorithm (a) the Reichardt motion detector (b) and theBarlow-Levick motion detector (c) (see Figure 9 for description of each panel) (see Supplementary Video S5 for full video)

Figure 14 Phase-based motion detection applied on the ldquohighwayvideordquo with added Gaussian white noise (see Supplementary VideoS6 for full video)

phase in the frequency domain that is due to rigid motionthe phase-based motion detection algorithm also incorpo-rates the Radon transform a transform closely related tothe Fourier transform Based on the Radon transform amotion indicator was proposed to robustly detect whetherthe phase change is caused by motion Therefore the algo-rithm can be efficiently implemented whenever the FFT isavailablesupported We showed examples of applying thephase-based motion detection algorithm on several videosequences We also showed that the locally detected motioncan be used for segmenting moving objects in video scenesWe compared the segmentation of moving objects using ourlocal phase-based algorithm to segmentation achieved usinga widely used optic flow based algorithm Our results suggestthat spatial phase information may provide an efficientalternative to perform many visual tasks in silico as well as in

Figure 15 Phase-based motion detection applied on the ldquotrainstation videordquo with added Gaussian white noise (see SupplementaryVideo S7 for full video)

modeling in vivo biological vision systems This is consistentwith other recent findings [15]

Note that phase information has been used for solvingvarious visual tasks in the past In fact phase has beensuccessfully employed in optic flow algorithms [24] andimage registration for translation [25] both applied tomotionrelated tasks The phase-based optic flow algorithm appliesthe optic flow equation on the local phase of images ratherthan the intensity itself to achieve better resolution androbustness in estimating motion velocity The phase correla-tionmethod computes the normalized cross-power spectrumof two images to extract the phase difference It providesbetter accuracy and robustness as compared to the classicalcross correlation approach applied to two consecutive imagesin a sequence However it has limitations when dealing withimages with a repetitive structure

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 17: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 17

(a) (b)

Figure 16 Segmentation of moving cars based on detected motion in the ldquohighway videordquo (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S8a and S8b for full videos)

(a) (b)

Figure 17 Segmentation of moving people in the ldquotrain station videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S9a and S9b for full videos)

(a) (b)

Figure 18 Segmentation of moving people in the ldquothermal videordquo based on detected motion (a) Motion detected by the phase-based motiondetection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S10a and S10b for full videos)

Our method differs from the above two cases inthe following ways First it employs a simple temporalderivativehigh-pass filtering on the phase to extract localphase changes The structure of the phase change is exploitedfor better detection of motion On the contrary the phasecorrelation method considers the structure of the phase

itself This also allowed us to make the motion detection amore local continuous process rather than purely operatingglobally on discrete frames Second it explores the structureof the change of phase due to motion in the frequencydomain rather than in the spatial domain in which thekey constraints of optic flow equations are based Third

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 18: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

18 Computational Intelligence and Neuroscience

(a) (b)

Figure 19 Segmentation of moving cars in the ldquowinter street videordquo based on detected motion (a) Motion detected by the phase-basedmotion detection algorithm (b) Motion detected using optic flow algorithm [4] (see Supplementary Videos S11a and S11b for full videos)

instead of focusing on estimating the exact velocity or shiftour method is centered on the detection of motion with acoarse local estimate of direction This is the case in the firststeps of biological motion detection in the retina or opticlobe of insects we discussed the resemblance of the methodpresented here to those of biological models of motiondetection

The proposed motion detection algorithm howevershares several advantages with the other phase-based meth-ods For example it is sensitive to motion that only induces asubpixel shift between frames and for very small differencesin intensity In addition when compared to the amplitude thelocal phase is robust under different contrast and illuminationconditions Consequently the algorithm presented here canoperate in a wide range of contrastillumination conditions

Furthermore once the local phase is extracted fromeach block motion detection becomes a localized temporaloperation on the local phase of each block This formsthe basis for the highly parallel structure in the phase-based motion detection algorithm By contrast traditionaloptic flow techniques often rely on explicit comparisonsacross spatial locations which increase for examplememorycomplexity since all states must be made available to theirneighbors

We also notice that for each block a large number ofmea-surements of phase changes are obtained From Figure 6 wesee that this number is much higher than that of the originalpixel space In other words in order to detect motion in arobust way the local phase-basedmotion detection algorithmundergoes an expansion of measurements before settlingdown onto a single motion indicator value This number ofmeasurements however does not incur additional computa-tional demand thanks to the highly efficient FFT algorithmBiological visual systems often have a similar structure Forexample in the vertebrate retina computations are carriedout by an extraordinarily large number of neurons untilmeasurements of the visual scene are projected by a fractionof the neurons onto the cortex [26] Similar expansion takesplace in the primary visual cortex as well

We also highlight the ease of implementation of theintrinsically parallel algorithm proposed here The algorithmintroduced in Section 33 is based on the FFT algorithmand does not require solving an optimization problem It

can be efficiently implemented in hardware for exampleFPGAs or in software for example using GPU acceleratorsWe note that extending the FFT to higher dimensions isstraightforward and the implementations of higher dimen-sional FFTs are also highly efficient Clearly themethodologycan be applied to motion detection of data in 3D or higherdimensional space where the Radon transform operates overplanes or hyperplanes

Finally we argue that the change of phase althoughhighly nonlinear can be obtained through a normalization(gain control) followed by a second order Volterra kernelThis separation of a higher order nonlinearity into gaincontrol block and a lower order nonlinear filter can be usedfor modeling motion detection circuits in biological systems

Appendix

The Derivative of the Local Phase

Let

119886 (120596119909 120596119910 119905)

= int

R2119906 (119909 119910 119905) 119908 (119909 119910) cos (120596

119909119909 + 120596119910119910) 119889119909 119889119910

119887 (120596119909 120596119910 119905)

= minusint

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

(A1)

and therefore

11986000(120596119909 120596119910 119905) 11989011989512060100(120596119909120596119910119905)

= 119886 (120596119909 120596119910 119905) + 119895119887 (120596

119909 120596119910 119905)

(A2)

The local phase amounts to 12060100(120596119909 120596119910 119905) = arctan 2(119887(120596

119909

120596119910 119905) 119886(120596

119909 120596119910 119905)) and therefore

11988912060100

119889119905

=

119886 (119889119887119889119905) minus 119887 (119889119886119889119905)

[119886]2+ [119887]2

(A3)

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 19: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Computational Intelligence and Neuroscience 19

Now

119889119886

119889119905

(120596119909 120596119910 119905)

= int

R2

120597119906

120597119905

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

(A4)

For a purely translational signal with instantaneous spatialshift given by (119904

119909(119905) 119904119910(119905)) within the support of the window

we have

119889119886

119889119905

(120596119909 120596119910 119905)

= minusint

R2

119889119904119909

119889119905

(119905)

120597119906

120597119909

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus int

R2

119889119904119910

119889119905

(119905)

120597119906

120597119910

(119909 119910 119905) 119908 (119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119909

119889119904119909

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

minus 120596119910

119889119904119910

119889119905

int

R2119906 (119909 119910 119905) 119908 (119909 119910) sin (120596

119909119909 + 120596119910119910) 119889119909 119889119910

=

119889119904119909

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199033=1199033(120596119909120596119910119905)

+ 120596119909

119889119904119909

119889119905

119887

+

119889119904119910

119889119905

int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) cos (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199034=1199034(120596119909120596119910119905)

+ 120596119910

119889119904119910

119889119905

119887

(A5)

Similarly

119889119887

119889119905

(120596119909 120596119910 119905) = minus

119889119904119909

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119909

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199035=1199035(120596119909120596119910119905)

minus 120596119909

119889119904119909

119889119905

119886 minus

119889119904119910

119889119905

sdot int

R2119906 (119909 119910 119905)

120597119908

120597119910

(119909 119910) sin (120596119909119909 + 120596119910119910) 119889119909 119889119910

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

1199036=1199036(120596119909120596119910119905)

minus 120596119910

119889119904119910

119889119905

119886

(A6)

Plugging in the value of 119889119886119889119905 and 119889119887119889119905 the derivative of thelocal phase amounts to

11988912060100

119889119905

(120596119909 120596119910 119905) =

minus (119889119904119909119889119905) [119903

3119887 + 1199035119886 + 120596119909 [

119886]2+ 120596119909 [

119887]2] minus (119889119904

119910119889119905) [119903

4119887 + 1199036119886 + 120596119910 [

119886]2+ 120596119910 [

119887]2]

[119886]2+ [119887]2

= minus

119889119904119909

119889119905

120596119909minus

119889119904119910

119889119905

120596119910minus

119889119904119909

119889119905

sdot

1199033119887 + 1199035119886

[119886]2+ [119887]2minus

119889119904119910

119889119905

sdot

1199034119887 + 1199036119886

[119886]2+ [119887]2

⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟⏟

v00(120596119909120596119910119905)

(A7)

Disclosure

The authorrsquos names are alphabetically listed

Conflict of Interests

The authors declare that there is no conflict of interestsregarding the publication of this paper

Acknowledgment

The research reported here was supported by AFOSR underGrant no FA9550-12-1-0232

References

[1] D Marr Vision A Computational Investigation into the HumanRepresentation and Processing of Visual Information MIT Press2010

[2] J R Sanes and S L Zipursky ldquoDesign principles of insect andvertebrate visual systemsrdquoNeuron vol 66 no 1 pp 15ndash36 2010

[3] A Borst ldquoIn search of the holy grail of fly motion visionrdquoEuropean Journal of Neuroscience vol 40 no 9 pp 3285ndash32932014

[4] D Sun S Roth and M J Black ldquoSecrets of optical flowestimation and their principlesrdquo in Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (CVPRrsquo10) pp 2432ndash2439 IEEE San Francisco Calif USA June 2010

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 20: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

20 Computational Intelligence and Neuroscience

[5] F Raudies ldquoOptic flowrdquo Scholarpedia vol 8 no 7 Article ID30724 2013

[6] G Orchard and R Etienne-Cummings ldquoBioinspired visualmotion estimationrdquo Proceedings of the IEEE vol 102 no 10 pp1520ndash1536 2014

[7] B Hassenstein and W Reichardt ldquoSystemtheoretische analyseder zeit- reihenfolgen-und vorzeichenauswertung bei der bewe-gungsperzeption des russelkafers chlorophanusrdquo Zeitschrift furNaturforschung B vol 11 no 9-10 pp 513ndash524 1956

[8] A Borst J Haag and D F Reiff ldquoFly motion visionrdquo AnnualReview of Neuroscience vol 33 pp 49ndash70 2010

[9] E H Adelson and J R Bergen ldquoSpatiotemporal energy modelsfor the perception of motionrdquo Journal of the Optical Society ofAmerica A Optics and Image Science and Vision vol 2 no 2pp 284ndash299 1985

[10] H B Barlow andW R Levick ldquoThemechanism of directionallyselective units in rabbitrsquos retinardquo The Journal of Physiology vol178 no 3 pp 477ndash504 1965

[11] A V Oppenheim and J S Lim ldquoThe importance of phase insignalsrdquo Proceedings of the IEEE vol 69 no 5 pp 529ndash541 1981

[12] T Gerkmann M Krawczyk-Becker and J Le Roux ldquoPhaseprocessing for single-channel speech enhancement history andrecent advancesrdquo IEEE Signal Processing Magazine vol 32 no2 pp 55ndash66 2015

[13] M Kadiri M Djebbouri and P Carre ldquoMagnitude-phase ofthe dual-tree quaternionic wavelet transform for multispectralsatellite image denoisingrdquoEURASIP Journal on Image andVideoProcessing vol 2014 no 1 article 41 16 pages 2014

[14] P Kovesi ldquoImage features from phase congruencyrdquo VidereJournal of Computer Vision Research vol 1 no 3 1999

[15] E Gladilin and R Eils ldquoOn the role of spatial phase andphase correlation in vision illusion and cognitionrdquo Frontiers inComputational Neuroscience vol 9 no 45 pp 1ndash14 2015

[16] P Dayan and L F Abbott Theoretical Neuroscience Computa-tional andMathematicalModeling of Neural SystemsMIT PressCambridge Mass USA 2001

[17] T S Lee ldquoImage representation using 2d gabor waveletsrdquo IEEETransactions on Pattern Analysis and Machine Intelligence vol18 no 10 pp 959ndash971 1996

[18] M H Hayes J S Lim and A V Oppenheim ldquoSignal recon-struction from phase or magnituderdquo IEEE Transactions onAcoustics Speech and Signal Processing vol 28 no 6 pp 672ndash680 1980

[19] E J Candes T Strohmer and V Voroninski ldquoPhaseLift exactand stable signal recovery from magnitude measurements viaconvex programmingrdquo Communications on Pure and AppliedMathematics vol 66 no 8 pp 1241ndash1274 2013

[20] A A Lazar E A Pnevmatikakis and Y Zhou ldquoThe powerof connectivity identity preserving transformations on visualstreams in the spike domainrdquo Neural Networks vol 44 pp 22ndash35 2013

[21] AA Lazar andY Zhou ldquoVolterra dendritic stimulus processorsand biophysical spike generators with intrinsic noise sourcesrdquoFrontiers in Computational Neuroscience vol 8 article 95 2014

[22] Y Wang P-M Jodoin F Porikli J Konrad Y Benezeth and PIshwar ldquoCDnet 2014 an expanded change detection benchmarkdatasetrdquo in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition Workshops (CVPRW rsquo14) pp393ndash400 IEEE Columbus Ohio USA June 2014

[23] A Klockner N Pinto Y Lee B Catanzaro P Ivanov and AFasih ldquoPyCUDA and PyOpenCL a scripting-based approach

to GPU run-time code generationrdquo Parallel Computing vol 38no 3 pp 157ndash174 2012

[24] D J Fleet and A D Jepson ldquoComputation of component imagevelocity from local phase informationrdquo International Journal ofComputer Vision vol 5 no 1 pp 77ndash104 1990

[25] E De Castro and C Morandi ldquoRegistration of translated androtated images using finite fourier transformsrdquo IEEE Transac-tions on Pattern Analysis and Machine Intelligence vol 9 no 5pp 700ndash703 1987

[26] C-J Jeon E Strettoi and R H Masland ldquoThe major cellpopulations of the mouse retinardquo The Journal of Neurosciencevol 18 no 21 pp 8936ndash8946 1998

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Page 21: Research Article A Motion Detection Algorithm Using Local ...bionet.github.io/papers/lazar_cin15.pdf · Research Article A Motion Detection Algorithm Using Local Phase Information

Submit your manuscripts athttpwwwhindawicom

Computer Games Technology

International Journal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Distributed Sensor Networks

International Journal of

Advances in

FuzzySystems

Hindawi Publishing Corporationhttpwwwhindawicom

Volume 2014

International Journal of

ReconfigurableComputing

Hindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

Artificial Intelligence

HindawithinspPublishingthinspCorporationhttpwwwhindawicom Volumethinsp2014

Advances inSoftware EngineeringHindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporation

httpwwwhindawicom Volume 2014

Advances in

Multimedia

International Journal of

Biomedical Imaging

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

ArtificialNeural Systems

Advances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

RoboticsJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Computational Intelligence and Neuroscience

Industrial EngineeringJournal of

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Modelling amp Simulation in EngineeringHindawi Publishing Corporation httpwwwhindawicom Volume 2014

The Scientific World JournalHindawi Publishing Corporation httpwwwhindawicom Volume 2014

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014

Human-ComputerInteraction

Advances in

Computer EngineeringAdvances in

Hindawi Publishing Corporationhttpwwwhindawicom Volume 2014